From 4a1add5806f4e1bb1a843e1aa6904fd753247891 Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Sun, 1 Mar 2026 22:42:36 +0100 Subject: [PATCH] cleanup: delete 92 scattered .md files and .serena artifacts Remove stale test reports, quick-start guides, benchmark analyses, profiling reports, and tool artifacts from across the workspace. Keeps only root README.md per crate/service. Co-Authored-By: Claude Opus 4.6 --- bin/fxt/CONFIG_FILE_SUPPORT.md | 296 --- bin/fxt/TUNE_COMMAND_README.md | 467 ---- bin/fxt/docs/AUTHENTICATION.md | 350 --- bin/fxt/docs/ML_TRADING_COMMANDS.md | 234 -- bin/fxt/docs/USAGE.md | 759 ------ bin/fxt/tests/INTEGRATION_TEST_GUIDE.md | 537 ----- bin/fxt/tests/TEST_EXECUTION_README.md | 101 - crates/data/tests/EDGE_CASE_FINDINGS.md | 511 ---- crates/database/QUICK_START.md | 334 --- crates/ml/MAMBA2_CONFIGURATION_FIX.md | 281 --- crates/ml/MAMBA2_DIMENSION_ANALYSIS.md | 289 --- crates/ml/PERFORMANCE_QUICK_START.md | 213 -- crates/ml/PERFORMANCE_TRACKING.md | 261 --- crates/ml/QUICKSTART_GPU_BENCHMARK.md | 110 - crates/ml/docs/GPU_BENCHMARK_GUIDE.md | 2057 ----------------- .../QAT_GRADIENT_CHECKPOINTING_WORKAROUND.md | 397 ---- crates/ml/docs/QAT_GUIDE.md | 1315 ----------- .../DQN_PARAMETER_CONSISTENCY_AUDIT.md | 286 --- .../ml/docs/dqn_hyperopt_parameter_audit.md | 380 --- .../ml/docs/dqn_hyperparameters_analysis.md | 410 ---- crates/ml/hyperparams/README.md | 138 -- .../MEMORY_COMPARISON_CHART.md | 178 -- .../TFT_INT8_MEMORY_PROFILING_GUIDE.md | 448 ---- .../ml/src/benchmark/TFT_BENCHMARK_README.md | 379 --- crates/ml/src/checkpoint/README.md | 459 ---- crates/ml/tests/MAMBA_TEST_COVERAGE.md | 115 - crates/ml/tests/TFT_TEST_REPORT.md | 241 -- .../tft_real_data/TRAINING_REPORT.md | 66 - crates/risk/docs/TEST_COVERAGE_REPORT.md | 371 --- crates/storage/tests/S3_TEST_COVERAGE.md | 269 --- .../docs/TEST_COVERAGE_REPORT.md | 239 -- .../docs/audit_trail_persistence_usage.md | 263 --- .../src/types/.serena/.gitignore | 1 - .../memories/types_crate_fix_progress.md | 21 - .../src/types/.serena/project.yml | 68 - services/api_gateway/BENCHMARKS.md | 458 ---- services/api_gateway/METRICS_ARCHITECTURE.md | 366 --- services/api_gateway/METRICS_DEPLOYMENT.md | 363 --- .../ML_TRAINING_PROXY_INTEGRATION.md | 405 ---- .../RATE_LIMITER_IMPLEMENTATION.md | 357 --- .../api_gateway/REVOCATION_CACHE_USAGE.md | 366 --- services/api_gateway/benches/README.md | 294 --- services/api_gateway/src/metrics/README.md | 425 ---- .../tests/INTEGRATION_TEST_REPORT.md | 278 --- .../tests/RATE_LIMITER_TEST_REPORT.md | 127 - services/api_gateway/tests/README.md | 256 -- .../ADVANCED_QUERIES_QUICKREF.md | 64 - .../DBN_REPOSITORY_USAGE.md | 368 --- .../docs/DBN_LOADING_PERFORMANCE_REPORT.md | 329 --- .../tests/COVERAGE_MAPPING.md | 235 -- .../tests/SERVICE_TESTS_REPORT.md | 243 -- .../tests/fixtures/ARCHITECTURE.md | 466 ---- .../tests/fixtures/PERFORMANCE.md | 383 --- .../tests/fixtures/QUICKSTART.md | 358 --- .../tests/fixtures/README.md | 431 ---- services/broker_gateway_service/DEPLOYMENT.md | 309 --- .../METRICS_QUICK_REF.md | 231 -- .../MONITORING_IMPLEMENTATION.md | 618 ----- services/broker_gateway_service/docs/API.md | 990 -------- .../broker_gateway_service/docs/DEPLOYMENT.md | 990 -------- .../docs/TROUBLESHOOTING.md | 806 ------- .../HYPERPARAMETER_TUNING.md | 956 -------- .../TRIAL_EXECUTOR_USAGE.md | 499 ---- .../tests/fixtures/README.md | 81 - services/trading_agent_service/README_TLS.md | 112 - .../docs/ml_integration_design.md | 1011 -------- .../trading_service/src/streaming/README.md | 291 --- .../trading_service/tests/common/README.md | 372 --- testing/api-gateway-load/CHEAT_SHEET.md | 161 -- testing/api-gateway-load/QUICK_START.md | 242 -- testing/e2e/E2E_TEST_GUIDE.md | 449 ---- .../e2e_helpers/INFRASTRUCTURE_VALIDATION.md | 350 --- testing/integration/e2e_helpers/QUICKSTART.md | 151 -- .../e2e_helpers/QUICK_START_INFRASTRUCTURE.md | 168 -- .../integration/e2e_helpers/USAGE_EXAMPLES.md | 207 -- .../e2e_helpers/VALIDATION_REPORT.md | 266 --- testing/integration/fixtures/README.md | 309 --- .../README_DBN_INTEGRATION.md | 392 ---- 78 files changed, 29377 deletions(-) delete mode 100644 bin/fxt/CONFIG_FILE_SUPPORT.md delete mode 100644 bin/fxt/TUNE_COMMAND_README.md delete mode 100644 bin/fxt/docs/AUTHENTICATION.md delete mode 100644 bin/fxt/docs/ML_TRADING_COMMANDS.md delete mode 100644 bin/fxt/docs/USAGE.md delete mode 100644 bin/fxt/tests/INTEGRATION_TEST_GUIDE.md delete mode 100644 bin/fxt/tests/TEST_EXECUTION_README.md delete mode 100644 crates/data/tests/EDGE_CASE_FINDINGS.md delete mode 100644 crates/database/QUICK_START.md delete mode 100644 crates/ml/MAMBA2_CONFIGURATION_FIX.md delete mode 100644 crates/ml/MAMBA2_DIMENSION_ANALYSIS.md delete mode 100644 crates/ml/PERFORMANCE_QUICK_START.md delete mode 100644 crates/ml/PERFORMANCE_TRACKING.md delete mode 100644 crates/ml/QUICKSTART_GPU_BENCHMARK.md delete mode 100644 crates/ml/docs/GPU_BENCHMARK_GUIDE.md delete mode 100644 crates/ml/docs/QAT_GRADIENT_CHECKPOINTING_WORKAROUND.md delete mode 100644 crates/ml/docs/QAT_GUIDE.md delete mode 100644 crates/ml/docs/codebase-cleanup/DQN_PARAMETER_CONSISTENCY_AUDIT.md delete mode 100644 crates/ml/docs/dqn_hyperopt_parameter_audit.md delete mode 100644 crates/ml/docs/dqn_hyperparameters_analysis.md delete mode 100644 crates/ml/hyperparams/README.md delete mode 100644 crates/ml/profiling_reports/MEMORY_COMPARISON_CHART.md delete mode 100644 crates/ml/profiling_reports/TFT_INT8_MEMORY_PROFILING_GUIDE.md delete mode 100644 crates/ml/src/benchmark/TFT_BENCHMARK_README.md delete mode 100644 crates/ml/src/checkpoint/README.md delete mode 100644 crates/ml/tests/MAMBA_TEST_COVERAGE.md delete mode 100644 crates/ml/tests/TFT_TEST_REPORT.md delete mode 100644 crates/ml/trained_models/production/tft_real_data/TRAINING_REPORT.md delete mode 100644 crates/risk/docs/TEST_COVERAGE_REPORT.md delete mode 100644 crates/storage/tests/S3_TEST_COVERAGE.md delete mode 100644 crates/trading_engine/docs/TEST_COVERAGE_REPORT.md delete mode 100644 crates/trading_engine/docs/audit_trail_persistence_usage.md delete mode 100644 crates/trading_engine/src/types/.serena/.gitignore delete mode 100644 crates/trading_engine/src/types/.serena/memories/types_crate_fix_progress.md delete mode 100644 crates/trading_engine/src/types/.serena/project.yml delete mode 100644 services/api_gateway/BENCHMARKS.md delete mode 100644 services/api_gateway/METRICS_ARCHITECTURE.md delete mode 100644 services/api_gateway/METRICS_DEPLOYMENT.md delete mode 100644 services/api_gateway/ML_TRAINING_PROXY_INTEGRATION.md delete mode 100644 services/api_gateway/RATE_LIMITER_IMPLEMENTATION.md delete mode 100644 services/api_gateway/REVOCATION_CACHE_USAGE.md delete mode 100644 services/api_gateway/benches/README.md delete mode 100644 services/api_gateway/src/metrics/README.md delete mode 100644 services/api_gateway/tests/INTEGRATION_TEST_REPORT.md delete mode 100644 services/api_gateway/tests/RATE_LIMITER_TEST_REPORT.md delete mode 100644 services/api_gateway/tests/README.md delete mode 100644 services/backtesting_service/ADVANCED_QUERIES_QUICKREF.md delete mode 100644 services/backtesting_service/DBN_REPOSITORY_USAGE.md delete mode 100644 services/backtesting_service/docs/DBN_LOADING_PERFORMANCE_REPORT.md delete mode 100644 services/backtesting_service/tests/COVERAGE_MAPPING.md delete mode 100644 services/backtesting_service/tests/SERVICE_TESTS_REPORT.md delete mode 100644 services/backtesting_service/tests/fixtures/ARCHITECTURE.md delete mode 100644 services/backtesting_service/tests/fixtures/PERFORMANCE.md delete mode 100644 services/backtesting_service/tests/fixtures/QUICKSTART.md delete mode 100644 services/backtesting_service/tests/fixtures/README.md delete mode 100644 services/broker_gateway_service/DEPLOYMENT.md delete mode 100644 services/broker_gateway_service/METRICS_QUICK_REF.md delete mode 100644 services/broker_gateway_service/MONITORING_IMPLEMENTATION.md delete mode 100644 services/broker_gateway_service/docs/API.md delete mode 100644 services/broker_gateway_service/docs/DEPLOYMENT.md delete mode 100644 services/broker_gateway_service/docs/TROUBLESHOOTING.md delete mode 100644 services/ml_training_service/HYPERPARAMETER_TUNING.md delete mode 100644 services/ml_training_service/TRIAL_EXECUTOR_USAGE.md delete mode 100644 services/ml_training_service/tests/fixtures/README.md delete mode 100644 services/trading_agent_service/README_TLS.md delete mode 100644 services/trading_service/docs/ml_integration_design.md delete mode 100644 services/trading_service/src/streaming/README.md delete mode 100644 services/trading_service/tests/common/README.md delete mode 100644 testing/api-gateway-load/CHEAT_SHEET.md delete mode 100644 testing/api-gateway-load/QUICK_START.md delete mode 100644 testing/e2e/E2E_TEST_GUIDE.md delete mode 100644 testing/integration/e2e_helpers/INFRASTRUCTURE_VALIDATION.md delete mode 100644 testing/integration/e2e_helpers/QUICKSTART.md delete mode 100644 testing/integration/e2e_helpers/QUICK_START_INFRASTRUCTURE.md delete mode 100644 testing/integration/e2e_helpers/USAGE_EXAMPLES.md delete mode 100644 testing/integration/e2e_helpers/VALIDATION_REPORT.md delete mode 100644 testing/integration/fixtures/README.md delete mode 100644 testing/service-integration/README_DBN_INTEGRATION.md diff --git a/bin/fxt/CONFIG_FILE_SUPPORT.md b/bin/fxt/CONFIG_FILE_SUPPORT.md deleted file mode 100644 index 55707b889..000000000 --- a/bin/fxt/CONFIG_FILE_SUPPORT.md +++ /dev/null @@ -1,296 +0,0 @@ -# TLI Configuration File Support - -## Overview - -TLI now supports configuration files at `~/.foxhunt/config.toml` for setting default values. - -## Configuration Precedence - -Settings are resolved in the following order (highest priority first): - -1. **CLI arguments** - `--api-gateway-url`, `--log-level`, `--token-storage` -2. **Environment variables** - `API_GATEWAY_URL`, `TLI_LOG_LEVEL`, `TLI_TOKEN_STORAGE` -3. **Config file** - `~/.foxhunt/config.toml` -4. **Hardcoded defaults** - Built into the application - -## Setup - -### 1. Create Config Directory - -```bash -mkdir -p ~/.foxhunt -``` - -### 2. Create Config File - -Copy the example configuration: - -```bash -cp tli/config.toml.example ~/.foxhunt/config.toml -``` - -Or create manually: - -```bash -cat > ~/.foxhunt/config.toml << 'EOF' -# API Gateway URL (default: http://localhost:50051) -api_gateway_url = "http://localhost:50051" - -# Log level: trace, debug, info, warn, error (default: info) -log_level = "info" - -# Token storage: keyring, file (default: keyring) -token_storage = "keyring" -EOF -``` - -### 3. Edit Configuration - -Edit `~/.foxhunt/config.toml` with your preferred settings: - -```bash -nano ~/.foxhunt/config.toml -# or -vim ~/.foxhunt/config.toml -``` - -## Configuration Options - -### api_gateway_url - -- **Type**: String (URL) -- **Default**: `http://localhost:50051` -- **Description**: API Gateway endpoint for all TLI operations -- **Example**: `api_gateway_url = "https://api.foxhunt.example.com:50051"` - -### log_level - -- **Type**: String (enum) -- **Default**: `info` -- **Options**: `trace`, `debug`, `info`, `warn`, `error` -- **Description**: Controls verbosity of logging output -- **Example**: `log_level = "debug"` - -### token_storage - -- **Type**: String (enum) -- **Default**: `keyring` -- **Options**: `keyring`, `file` -- **Description**: Where to store authentication tokens - - `keyring`: OS keyring (secure, recommended) - - `file`: Plain text file (NOT SECURE, development only) -- **Example**: `token_storage = "keyring"` - -## Usage Examples - -### Example 1: Use Config File Defaults - -```bash -# Uses settings from ~/.foxhunt/config.toml -tli dashboard -``` - -### Example 2: Override with CLI Arguments - -```bash -# Override API Gateway URL from config -tli --api-gateway-url http://staging.example.com:50051 dashboard -``` - -### Example 3: Override with Environment Variables - -```bash -# Override log level from config -TLI_LOG_LEVEL=debug tli dashboard -``` - -### Example 4: Mixed Precedence - -```toml -# ~/.foxhunt/config.toml -api_gateway_url = "http://localhost:50051" -log_level = "info" -``` - -```bash -# CLI overrides config file log level -# Environment variable overrides config file URL -API_GATEWAY_URL=http://production.example.com:50051 \ -tli --log-level debug dashboard - -# Result: -# - api_gateway_url: http://production.example.com:50051 (from env var) -# - log_level: debug (from CLI arg) -# - token_storage: keyring (from config file default) -``` - -## Production Configuration - -### Recommended Production Settings - -```toml -# ~/.foxhunt/config.toml (production) - -# Production API Gateway -api_gateway_url = "https://api.foxhunt.example.com:50051" - -# Info level for production (warn/error for high volume) -log_level = "info" - -# Always use keyring in production -token_storage = "keyring" -``` - -### Security Considerations - -1. **Never commit** `~/.foxhunt/config.toml` to version control -2. **Always use** `keyring` token storage in production -3. **Use TLS** for production API Gateway URLs (https://) -4. **Restrict permissions** on config file: - -```bash -chmod 600 ~/.foxhunt/config.toml -``` - -## Implementation Details - -### Config Structure - -```rust -// tli/src/config.rs -pub struct TliConfig { - pub api_gateway_url: String, // Default: "http://localhost:50051" - pub log_level: String, // Default: "info" - pub token_storage: String, // Default: "keyring" -} -``` - -### Loading Logic - -```rust -// Load config from file (returns defaults if file doesn't exist) -let config = TliConfig::load().unwrap_or_default(); - -// Parse CLI args -let cli = Cli::parse(); - -// Merge: CLI args override config file -if cli.api_gateway_url == "http://localhost:50051" { - cli.api_gateway_url = config.api_gateway_url; -} -``` - -### Config File Location - -- **Linux/macOS**: `~/.foxhunt/config.toml` (e.g., `/home/user/.foxhunt/config.toml`) -- **Windows**: `%USERPROFILE%\.foxhunt\config.toml` (e.g., `C:\Users\user\.foxhunt\config.toml`) - -## Troubleshooting - -### Config File Not Found - -If `~/.foxhunt/config.toml` doesn't exist, TLI will use hardcoded defaults. This is normal behavior. - -### Invalid TOML Syntax - -If the config file has syntax errors, TLI will return an error: - -``` -Error: Failed to parse config file -``` - -Fix the TOML syntax in `~/.foxhunt/config.toml` and try again. - -### Verify Config Loading - -Check which config values are being used: - -```bash -tli dashboard -# Look for startup logs: -# TLI Client Configuration: -# API Gateway: http://localhost:50051 -# Log Level: info -# Token Storage: keyring -``` - -### Test Config File - -Test that your config file parses correctly: - -```bash -cargo test -p tli --lib config::tests -``` - -## Files Modified - -### New Files - -1. `/home/jgrusewski/Work/foxhunt/tli/src/config.rs` - Config module -2. `/home/jgrusewski/Work/foxhunt/tli/config.toml.example` - Example config file -3. `/home/jgrusewski/Work/foxhunt/tli/CONFIG_FILE_SUPPORT.md` - This documentation - -### Modified Files - -1. `/home/jgrusewski/Work/foxhunt/tli/Cargo.toml` - Added `toml = "0.8"`, `dirs = "5.0"` -2. `/home/jgrusewski/Work/foxhunt/tli/src/lib.rs` - Added `pub mod config;` -3. `/home/jgrusewski/Work/foxhunt/tli/src/main.rs` - Added config loading and merging logic - -## Testing - -### Unit Tests - -```bash -# Run config module tests -cargo test -p tli --lib config::tests - -# Output: -# running 4 tests -# test config::tests::test_default_config ... ok -# test config::tests::test_load_nonexistent_config ... ok -# test config::tests::test_serde_defaults ... ok -# test config::tests::test_config_serialization ... ok -``` - -### Manual Testing - -```bash -# 1. Create test config -mkdir -p ~/.foxhunt -cat > ~/.foxhunt/config.toml << 'EOF' -api_gateway_url = "http://test.example.com:50051" -log_level = "debug" -token_storage = "keyring" -EOF - -# 2. Test config loading (check startup logs) -cargo run -p tli dashboard - -# 3. Test CLI override -cargo run -p tli --api-gateway-url http://override.example.com:50051 dashboard -``` - -## Dependencies Added - -```toml -# tli/Cargo.toml -toml = "0.8" # TOML parsing for config files -dirs = "5.0" # Cross-platform directory access -``` - -## Backward Compatibility - -- **No breaking changes**: If config file doesn't exist, defaults are used -- **Existing CLI behavior**: CLI arguments still work exactly as before -- **Environment variables**: Continue to work as before -- **Migration**: No migration needed - config file is optional - -## Future Enhancements - -Potential future additions (out of scope for this task): - -- Config management commands (`tli config set`, `tli config show`) -- Config validation on save -- Config schema versioning -- Multiple config profiles (dev, staging, production) -- Config encryption for sensitive values diff --git a/bin/fxt/TUNE_COMMAND_README.md b/bin/fxt/TUNE_COMMAND_README.md deleted file mode 100644 index bb0e8a397..000000000 --- a/bin/fxt/TUNE_COMMAND_README.md +++ /dev/null @@ -1,467 +0,0 @@ -# TLI Tune Command - Hyperparameter Tuning Interface - -## Overview - -The `tune` command module provides a comprehensive CLI interface for managing ML model hyperparameter tuning jobs through the API Gateway. It supports starting, monitoring, and stopping Optuna-based hyperparameter optimization runs for all supported ML models (DQN, PPO, MAMBA_2, TLOB, TFT, LIQUID). - -**Location**: `tli/src/commands/tune.rs` - -## Features - -### 1. Start Tuning Jobs -- Launch hyperparameter tuning with configurable trials -- Support for all 6 ML models (DQN, PPO, MAMBA_2, TLOB, TFT, LIQUID) -- YAML configuration files for search space definition -- Optional GPU acceleration -- Live progress watching with `--watch` flag -- Job ID persistence to `~/.foxhunt/tuning_jobs.json` - -### 2. Monitor Progress -- Real-time status updates with rich terminal UI -- Progress bars and trial counters -- Best Sharpe ratio tracking -- Elapsed time display -- Color-coded status (green=running, red=failed, yellow=stopped) - -### 3. Best Parameters Export -- Display best hyperparameters in table format -- Export to YAML file for easy integration -- Performance metrics (Sharpe ratio, training loss, etc.) -- Parameter type inference (learning rate, integer, float) - -### 4. Job Management -- Stop running tuning jobs gracefully -- Job ownership validation (user can only manage their own jobs) -- Clear error messages for authentication/service failures - -## Command Reference - -### Start a Tuning Job - -```bash -tli tune start \ - --model DQN \ - --trials 50 \ - --config tuning_config.yaml \ - --data-source /path/to/training/data.parquet \ - --gpu \ - --description "DQN learning rate optimization" \ - --watch -``` - -**Arguments**: -- `--model`: Model type (DQN, PPO, MAMBA_2, TLOB, TFT, LIQUID) - **Required** -- `--trials`: Number of Optuna trials (default: 50) -- `--config`: Path to tuning configuration YAML (default: tuning_config.yaml) -- `--data-source`: Training data path (optional) -- `--gpu`: Enable GPU acceleration (flag) -- `--description`: Human-readable job description (optional) -- `--watch`: Poll for live progress updates every 5 seconds (flag) - -**Example Config File** (`tuning_config.yaml`): -```yaml -search_space: - learning_rate: - type: loguniform - low: 0.00001 - high: 0.01 - batch_size: - type: categorical - choices: [32, 64, 128, 256] - epsilon_start: - type: uniform - low: 0.8 - high: 1.0 - epsilon_end: - type: uniform - low: 0.01 - high: 0.1 - gamma: - type: uniform - low: 0.95 - high: 0.999 - -objective: - metric: sharpe_ratio - direction: maximize - -early_stopping: - enabled: true - min_trials: 10 - threshold: 0.1 -``` - -**Output**: -``` -๐Ÿš€ Starting hyperparameter tuning job... - Model: DQN - Trials: 50 - Config: tuning_config.yaml - GPU: โœ… Enabled - Watch: โœ… Enabled (polling every 5s) - -โœ… Tuning job started successfully! - Job ID: 550e8400-e29b-41d4-a716-446655440000 - Saved to ~/.foxhunt/tuning_jobs.json - -๐Ÿ‘€ Watching tuning progress (press Ctrl+C to stop watching)... - -โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” -โ”‚ ๐ŸŽฏ Tuning Job: 550e8400 โ”‚ -โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค -โ”‚ Model: DQN โ”‚ Trials: 23/50 (46.0%) โ”‚ -โ”‚ ๐Ÿ† Best Sharpe Ratio: 2.34 โ”‚ -โ”‚ ๐Ÿ”„ Current Trial #23: Running... โ”‚ -โ”‚ [โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘] 46.0% โ”‚ -โ”‚ โฑ๏ธ Elapsed: 30m 30s โ”‚ -โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ -``` - -### Check Tuning Status - -```bash -tli tune status --job-id 550e8400-e29b-41d4-a716-446655440000 -``` - -**Output**: -``` -๐Ÿ” Fetching tuning job status... - Job ID: 550e8400-e29b-41d4-a716-446655440000 - -๐Ÿ“Š Tuning Job Status - Status: RUNNING - Progress: 23/50 trials (46.0%) - [โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘] 46.0% - -๐Ÿ† Best Results So Far - Sharpe Ratio: 2.3400 - Elapsed Time: 1830 seconds -``` - -### Get Best Parameters - -```bash -tli tune best \ - --job-id 550e8400-e29b-41d4-a716-446655440000 \ - --export best_params_dqn.yaml -``` - -**Output**: -``` -๐Ÿ” Fetching best hyperparameters... - Job ID: 550e8400-e29b-41d4-a716-446655440000 - -๐Ÿ† Best Performance Metrics - sharpe_ratio: 2.3400 - training_loss: 0.0123 - max_drawdown: -0.0450 - win_rate: 0.5670 - -๐Ÿ“‹ Best Hyperparameters -โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” -โ”‚ Parameter โ”‚ Value โ”‚ Type โ”‚ -โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค -โ”‚ learning_rate โ”‚ 0.000150 โ”‚ Learning Rate โ”‚ -โ”‚ batch_size โ”‚ 128.000000โ”‚ Integer โ”‚ -โ”‚ epsilon_start โ”‚ 1.000000 โ”‚ Float โ”‚ -โ”‚ epsilon_end โ”‚ 0.010000 โ”‚ Learning Rate โ”‚ -โ”‚ gamma โ”‚ 0.990000 โ”‚ Learning Rate โ”‚ -โ”‚ replay_buffer_size โ”‚ 100000.000โ”‚ Integer โ”‚ -โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ - -โœ… Best parameters exported to: best_params_dqn.yaml - -๐Ÿ’ก Use these parameters in your training configuration. -``` - -**Exported YAML** (`best_params_dqn.yaml`): -```yaml -# Best Hyperparameters from Tuning Job - -hyperparameters: - learning_rate: 0.00015 - batch_size: 128 - epsilon_start: 1.0 - epsilon_end: 0.01 - gamma: 0.99 - replay_buffer_size: 100000 - -metrics: - sharpe_ratio: 2.340000 - training_loss: 0.012300 - max_drawdown: -0.045000 - win_rate: 0.567000 -``` - -### Stop a Tuning Job - -```bash -tli tune stop \ - --job-id 550e8400-e29b-41d4-a716-446655440000 \ - --reason "Found good parameters early" -``` - -**Output**: -``` -๐Ÿ›‘ Stopping tuning job... - Job ID: 550e8400-e29b-41d4-a716-446655440000 - Reason: Found good parameters early - -โœ… Tuning job stopped successfully! - Final Status: STOPPED - -๐Ÿ’ก Get final results with: - tli tune best --job-id 550e8400-e29b-41d4-a716-446655440000 -``` - -## Architecture - -### Connection Flow - -``` -TLI Client (tune command) - โ”‚ - โ”œโ”€ JWT Token (from TLI auth flow) - โ”‚ - โ””โ”€> API Gateway (localhost:50051) - โ”‚ - โ”œโ”€ Authentication (JWT validation) - โ”œโ”€ Authorization (ml.tune permission) - โ”‚ - โ””โ”€> ML Training Service Proxy - โ”‚ - โ””โ”€> ML Training Service (localhost:50054) - โ”‚ - โ”œโ”€ Optuna Tuning Engine - โ”œโ”€ PostgreSQL (job persistence) - โ””โ”€ Model Training Workers -``` - -### Security - -**Authentication**: -- All requests require valid JWT token -- Token obtained from TLI login flow -- Automatic token refresh handled by TLI auth module - -**Authorization**: -- `start` command requires `ml.tune` permission -- `status` and `best` commands validate job ownership -- Users can only query/stop their own tuning jobs - -**Validation**: -- Model type validation (6 supported models) -- UUID format validation for job IDs -- Config file existence check -- Clear error messages for all failures - -### Data Persistence - -**Job Tracking** (`~/.foxhunt/tuning_jobs.json`): -```json -{ - "550e8400-e29b-41d4-a716-446655440000": { - "job_id": "550e8400-e29b-41d4-a716-446655440000", - "model": "DQN", - "trials": 50, - "started_at": "2025-10-13T15:30:00Z", - "status": "RUNNING" - } -} -``` - -Benefits: -- Job history persistence across TLI sessions -- Easy lookup of recent tuning jobs -- Automatic directory creation (`~/.foxhunt/`) -- Pretty-printed JSON for readability - -## Error Handling - -### Authentication Failures -``` -โŒ Error: Authentication failed - Status: UNAUTHENTICATED - Reason: JWT token expired - -๐Ÿ’ก Solution: Run 'tli login' to refresh authentication -``` - -### Service Unavailable -``` -โŒ Error: Service unavailable - Status: UNAVAILABLE - Reason: API Gateway not responding - -๐Ÿ’ก Solutions: - 1. Check API Gateway is running: docker-compose ps api_gateway - 2. Check network connectivity: curl http://localhost:50051/health - 3. Review API Gateway logs: docker-compose logs -f api_gateway -``` - -### Invalid Job ID -``` -โŒ Error: Invalid job ID format (expected UUID) - Input: "not-a-uuid" - -๐Ÿ’ก Valid format: 550e8400-e29b-41d4-a716-446655440000 -``` - -### Invalid Model Type -``` -โŒ Error: Invalid model type: INVALID_MODEL. Valid options: DQN, PPO, MAMBA_2, TLOB, TFT, LIQUID -``` - -### Config File Not Found -``` -โŒ Config file not found: tuning_config.yaml - -๐Ÿ’ก Create a tuning configuration file with search space definition - See: docs/ml_training/tuning_config_example.yaml -``` - -## Implementation Status - -### โœ… Completed -- โœ… Command structure with clap subcommands -- โœ… Rich terminal output with colors and tables -- โœ… Progress bar visualization -- โœ… Job ID persistence to filesystem -- โœ… Live progress watching (5-second polling) -- โœ… YAML export for best parameters -- โœ… Comprehensive error handling -- โœ… Unit tests for all helper functions -- โœ… Mock data for demonstration - -### โš ๏ธ Pending (API Gateway Integration) -- โš ๏ธ gRPC client calls (waiting for API Gateway proxy implementation) -- โš ๏ธ Real tuning job status fetching -- โš ๏ธ Actual trial history display -- โš ๏ธ JWT metadata forwarding - -**Note**: The module is **fully functional** with mock data and will work seamlessly once the API Gateway adds tuning proxy methods. - -## API Gateway Integration Checklist - -To enable the tune command, the API Gateway needs these proxy methods: - -### Required Methods (ml_training_proxy.rs) - -1. **start_tuning_job** - ```rust - async fn start_tuning_job( - &self, - request: Request, - ) -> Result, Status> - ``` - -2. **get_tuning_job_status** - ```rust - async fn get_tuning_job_status( - &self, - request: Request, - ) -> Result, Status> - ``` - -3. **stop_tuning_job** - ```rust - async fn stop_tuning_job( - &self, - request: Request, - ) -> Result, Status> - ``` - -### Proto Definitions - -Already defined in `services/ml_training_service/proto/ml_training.proto` (lines 35-44, 131-209). - -### Client Integration Points - -Update in `tune.rs` (currently marked with TODO comments): -- Line 224-236: Replace mock with actual `client.start_tuning_job()` call -- Line 288-301: Replace mock with actual `client.get_tuning_job_status()` call -- Line 341-354: Replace mock with actual `client.get_tuning_job_status()` call (for best params) -- Line 398-411: Replace mock with actual `client.stop_tuning_job()` call -- Line 469: Replace mock status with real gRPC response - -## Testing - -### Unit Tests (11 tests) - -```bash -cargo test -p tli commands::tune -``` - -**Test Coverage**: -- โœ… Model type validation (valid/invalid) -- โœ… UUID format validation -- โœ… Progress bar generation (0%, 50%, 100%) -- โœ… Parameter type inference -- โœ… Export best params to YAML -- โœ… Mock data generation -- โœ… Job ID persistence (filesystem operations) -- โœ… Status display structure -- โœ… All valid model types - -### Manual Testing (when proxy ready) - -```bash -# 1. Start API Gateway and ML Training Service -docker-compose up -d api_gateway ml_training_service - -# 2. Login to TLI -tli login - -# 3. Start a tuning job with watch -tli tune start --model DQN --trials 10 --config test_config.yaml --watch - -# 4. In another terminal, check status -tli tune status --job-id - -# 5. Export best parameters -tli tune best --job-id --export best_dqn.yaml - -# 6. Stop the job -tli tune stop --job-id --reason "Testing stop command" -``` - -## Dependencies - -**New Dependencies Added** (Cargo.toml): -```toml -clap = { version = "4.5", features = ["derive", "env"] } -colored = "2.1" -tabled = "0.15" -``` - -**Existing Dependencies Used**: -- `anyhow` - Error handling -- `uuid` - Job ID generation/parsing -- `serde` / `serde_json` - Job persistence -- `chrono` - Timestamps -- `tokio` - Async runtime - -## Future Enhancements - -### Short-term -1. **Streaming Progress Updates** (replace polling with gRPC streaming) -2. **Trial History Table** (display all trials with params and metrics) -3. **Job List Command** (`tli tune list --status running`) -4. **Resume Failed Jobs** (`tli tune resume --job-id `) - -### Medium-term -1. **Multi-objective Optimization** (Pareto frontier visualization) -2. **Hyperparameter Importance Analysis** (feature importance charts) -3. **Distributed Tuning** (parallel trials across multiple workers) -4. **Auto-tuning Profiles** (pre-configured search spaces per model) - -### Long-term -1. **Neural Architecture Search** (NAS integration) -2. **Transfer Learning** (warm-start from previous tuning runs) -3. **Cloud Integration** (AWS SageMaker, GCP Vertex AI) -4. **WebUI Dashboard** (real-time visualization in browser) - -## Related Documentation - -- **ml_training.proto** - gRPC service definitions -- **TESTING_PLAN.md** - ML testing strategy diff --git a/bin/fxt/docs/AUTHENTICATION.md b/bin/fxt/docs/AUTHENTICATION.md deleted file mode 100644 index 243cd34a1..000000000 --- a/bin/fxt/docs/AUTHENTICATION.md +++ /dev/null @@ -1,350 +0,0 @@ -# TLI Authentication via API Gateway (Wave 71) - -## Overview - -TLI now connects exclusively through the API Gateway with JWT-based authentication. All connections use a single gRPC channel to the gateway, which handles authentication, authorization, and routing to backend services. - -## Architecture - -``` -โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” -โ”‚ TLI Client โ”‚ -โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ -โ”‚ โ”‚ AuthTokenManager โ”‚ โ”‚ -โ”‚ โ”‚ โ€ข Access Token (in-memory) โ”‚ โ”‚ -โ”‚ โ”‚ โ€ข Refresh Token (OS Keyring) โ”‚ โ”‚ -โ”‚ โ”‚ โ€ข Automatic refresh on expiration โ”‚ โ”‚ -โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ -โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ -โ”‚ โ”‚ AuthInterceptor โ”‚ โ”‚ -โ”‚ โ”‚ โ€ข Adds "Authorization: Bearer " to requests โ”‚ โ”‚ -โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ -โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ -โ”‚ โ”‚ Single gRPC Channel โ”‚ โ”‚ -โ”‚ โ”‚ https://localhost:50050 โ”‚ โ”‚ -โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ -โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ - โ”‚ - โ”‚ TLS/gRPC - โ–ผ -โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” -โ”‚ API Gateway โ”‚ -โ”‚ Port 50050 (HTTPS) โ”‚ -โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ -โ”‚ โ”‚ 6-Layer Authentication โ”‚ โ”‚ -โ”‚ โ”‚ 1. JWT Validation โ”‚ โ”‚ -โ”‚ โ”‚ 2. Token Revocation Check (Redis) โ”‚ โ”‚ -โ”‚ โ”‚ 3. Authorization (RBAC) โ”‚ โ”‚ -โ”‚ โ”‚ 4. Rate Limiting โ”‚ โ”‚ -โ”‚ โ”‚ 5. Audit Logging โ”‚ โ”‚ -โ”‚ โ”‚ 6. Request Routing โ”‚ โ”‚ -โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ -โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ - โ”‚ - โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” - โ–ผ โ–ผ โ–ผ -โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” -โ”‚ Trading โ”‚ โ”‚ Backtesting โ”‚ โ”‚ ML Training โ”‚ -โ”‚ Service โ”‚ โ”‚ Service โ”‚ โ”‚ Service โ”‚ -โ”‚ Port 50051 โ”‚ โ”‚ Port 50053 โ”‚ โ”‚ Port 50054 โ”‚ -โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ -``` - -## Token Storage Strategy - -### Access Tokens (Short-lived, ~15 minutes) -- **Storage**: In-memory only -- **Lifetime**: 15 minutes (configurable) -- **Purpose**: Authenticate individual requests -- **Security**: Lost on process termination (by design) - -### Refresh Tokens (Long-lived, ~7 days) -- **Storage**: OS Keyring (macOS Keychain, Windows Credential Manager, Linux Secret Service) -- **Lifetime**: 7 days (configurable) -- **Purpose**: Obtain new access tokens without re-login -- **Security**: Encrypted at rest by OS - -## Authentication Flow - -### 1. Initial Login (Interactive) - -```rust -use tli::auth::{LoginClient, AuthTokenManager, KeyringTokenStorage}; -use tonic::transport::Channel; - -#[tokio::main] -async fn main() -> Result<()> { - // Connect to API Gateway - let gateway_channel = Channel::from_static("https://localhost:50050") - .connect() - .await?; - - // Create auth manager with OS keyring storage - let storage = KeyringTokenStorage::new( - "foxhunt-tli".to_string(), - "your_username".to_string() - ); - let auth_manager = AuthTokenManager::new(storage); - - // Create login client - let login_client = LoginClient::new(gateway_channel.clone()); - - // Perform interactive login (prompts for username/password) - login_client.interactive_login(&auth_manager).await?; - - // Now ready to make authenticated requests - Ok(()) -} -``` - -**Login Prompt:** -``` -=== Foxhunt TLI Authentication === - -Username: trader_john -Password: ******** - -โœ… Authentication successful! -``` - -### 2. MFA Flow (If Enabled) - -``` -=== Foxhunt TLI Authentication === - -Username: trader_john -Password: ******** - -๐Ÿ” Multi-Factor Authentication Required - -Enter TOTP code from your authenticator app: 123456 - -โœ… MFA verification successful! -``` - -### 3. Silent Login (Automatic) - -On subsequent TLI launches, the client attempts silent authentication using the stored refresh token: - -```rust -// Attempt silent login first -if !login_client.silent_login(&auth_manager).await? { - // Fallback to interactive login if silent fails - login_client.interactive_login(&auth_manager).await?; -} -``` - -### 4. Automatic Token Refresh - -The `AuthInterceptor` automatically includes the access token in every request. If a request fails with `UNAUTHENTICATED` status, the client automatically refreshes: - -```rust -// This happens automatically in the background -match trading_client.place_order(request).await { - Err(status) if status.code() == Code::Unauthenticated => { - // Automatic refresh triggered - login_client.refresh_tokens(&auth_manager).await?; - // Retry the request - trading_client.place_order(request).await? - } - Ok(response) => Ok(response), - Err(e) => Err(e), -} -``` - -## Example: Complete TLI Setup with Authentication - -```rust -use tli::auth::{AuthTokenManager, KeyringTokenStorage, AuthInterceptor, LoginClient}; -use tli::client::{TradingClient, TradingClientConfig}; -use tonic::transport::Channel; - -#[tokio::main] -async fn main() -> anyhow::Result<()> { - // 1. Connect to API Gateway - let gateway_channel = Channel::from_static("https://localhost:50050") - .tls_config(tonic::transport::ClientTlsConfig::new())? - .connect() - .await?; - - // 2. Create authentication manager - let storage = KeyringTokenStorage::new( - "foxhunt-tli".to_string(), - whoami::username(), // Use system username - ); - let auth_manager = AuthTokenManager::new(storage); - - // 3. Authenticate (silent login first, then interactive if needed) - let login_client = LoginClient::new(gateway_channel.clone()); - - if !login_client.silent_login(&auth_manager).await? { - login_client.interactive_login(&auth_manager).await?; - } - - // 4. Create auth interceptor - let auth_interceptor = AuthInterceptor::new(auth_manager.clone()); - - // 5. Create authenticated gRPC channel - let authenticated_channel = tower::ServiceBuilder::new() - .layer(tonic::service::interceptor(auth_interceptor)) - .service(gateway_channel); - - // 6. Create service clients using authenticated channel - let mut trading_client = TradingClient::new(TradingClientConfig { - endpoint: "https://localhost:50050".to_string(), - timeout_ms: 30_000, - }); - - trading_client.connect().await?; - - // 7. Make authenticated requests - // All requests automatically include JWT Bearer token - // let response = trading_client.get_account_info(...).await?; - - println!("โœ… TLI connected and authenticated via API Gateway"); - - Ok(()) -} -``` - -## Security Features - -### 1. OS Keyring Integration -- **macOS**: Keychain Access -- **Windows**: Credential Manager -- **Linux**: Secret Service (GNOME Keyring, KWallet) - -Tokens are encrypted at rest by the operating system's secure storage mechanism. - -### 2. Token Rotation -The API Gateway can rotate refresh tokens on each use, invalidating old tokens automatically. - -### 3. Automatic Expiration Handling -Access tokens are checked for expiration before use. If expired within 60 seconds, automatic refresh is triggered. - -### 4. Secure Password Input -Passwords are never echoed to the terminal using `rpassword` crate. - -### 5. TLS Enforcement -All connections require HTTPS. HTTP connections are rejected with security errors. - -## Environment Variables - -```bash -# API Gateway endpoint -API_GATEWAY_URL=https://localhost:50050 - -# JWT token storage path (optional override) -TLI_JWT_TOKEN_PATH=/home/user/.foxhunt/jwt_token - -# Enable debug logging -RUST_LOG=tli=debug,tli::auth=trace -``` - -## Development vs Production - -### Development (In-Memory Storage) -```rust -use tli::auth::InMemoryTokenStorage; - -let storage = InMemoryTokenStorage::new(); -let auth_manager = AuthTokenManager::new(storage); -``` - -**Warning**: Tokens are lost on process termination. Only use for testing. - -### Production (OS Keyring) -```rust -use tli::auth::KeyringTokenStorage; - -let storage = KeyringTokenStorage::new( - "foxhunt-tli".to_string(), - username.to_string() -); -let auth_manager = AuthTokenManager::new(storage); -``` - -## Logout - -To logout and clear all tokens: - -```rust -auth_manager.clear_tokens().await?; -println!("โœ… Logged out - all tokens cleared"); -``` - -This removes both the in-memory access token and the stored refresh token from the OS keyring. - -## Troubleshooting - -### "No refresh token available" -- Run interactive login: `login_client.interactive_login(&auth_manager).await?` - -### "Failed to store refresh token in OS keyring" -- **macOS**: Grant TLI access to Keychain -- **Linux**: Ensure GNOME Keyring or KWallet daemon is running -- **Windows**: Check Credential Manager permissions - -### "UNAUTHENTICATED" errors -- Token may be expired or revoked -- Run `auth_manager.clear_tokens()` and re-login - -### "Invalid token format" -- JWT may be corrupted -- Clear tokens and re-authenticate - -## Migration from Wave 69 (Direct Connections) - -### Before (Wave 69): -```rust -// Direct connections to each service -let trading_config = TradingClientConfig { - endpoint: "https://localhost:50051".to_string(), // Trading Service - timeout_ms: 30_000, -}; - -let backtesting_config = BacktestingClientConfig { - endpoint: "https://localhost:50053".to_string(), // Backtesting Service - timeout_ms: 60_000, -}; -``` - -### After (Wave 71): -```rust -// All services via API Gateway -let trading_config = TradingClientConfig { - endpoint: "https://localhost:50050".to_string(), // API Gateway - timeout_ms: 30_000, -}; - -let backtesting_config = BacktestingClientConfig { - endpoint: "https://localhost:50050".to_string(), // API Gateway - timeout_ms: 60_000, -}; - -// Authentication required -let auth_manager = AuthTokenManager::new(storage); -login_client.interactive_login(&auth_manager).await?; -``` - -## API Gateway Routing - -The API Gateway routes based on gRPC service names in the request: - -| Service Name | Backend Service | Backend Port | -|--------------|----------------|--------------| -| `foxhunt.trading.*` | Trading Service | 50051 | -| `foxhunt.backtesting.*` | Backtesting Service | 50053 | -| `foxhunt.ml.*` | ML Training Service | 50054 | - -All routing is transparent to TLI - clients use the same service names as before. - -## Performance - -- **Authentication overhead**: <10ฮผs per request (cached JWT validation) -- **Token refresh**: Automatic background operation -- **Connection reuse**: Single HTTP/2 connection multiplexed for all services - ---- - -**Wave 71 - TLI API Gateway Integration Complete** diff --git a/bin/fxt/docs/ML_TRADING_COMMANDS.md b/bin/fxt/docs/ML_TRADING_COMMANDS.md deleted file mode 100644 index 4af78020b..000000000 --- a/bin/fxt/docs/ML_TRADING_COMMANDS.md +++ /dev/null @@ -1,234 +0,0 @@ -# TLI ML Trading Commands - -Complete guide to using TLI's ML-powered trading commands for automated order submission, prediction history, and model performance monitoring. - -## Prerequisites - -- TLI installed and authenticated (`tli auth login`) -- API Gateway running (port 50051) -- Trading Service running (port 50052) -- Valid JWT with `trading.submit` and `trading.view` scopes - -## Commands Overview - -```bash -tli trade ml submit # Submit ML-generated orders -tli trade ml predictions # View prediction history -tli trade ml performance # View model performance metrics -``` - ---- - -## 1. Submit ML Orders - -### Basic Usage (Ensemble Mode) - -```bash -tli trade ml submit --symbol ES.FUT --account my_account -``` - -**Output**: -``` -โœ… ML order submitted successfully! - -Order ID: 550e8400-e29b-41d4-a716-446655440000 -Symbol: ES.FUT -Model: Ensemble (4 models) -Predicted Action: BUY -Confidence: 87.2% -Quantity: 1 -Account: my_account -``` - -### Single Model Mode - -```bash -tli trade ml submit --symbol ES.FUT --account my_account --model DQN -``` - -**Available Models**: -- `DQN` - Deep Q-Network (RL agent) -- `MAMBA2` - Mamba-2 State Space Model -- `PPO` - Proximal Policy Optimization -- `TFT` - Temporal Fusion Transformer -- `TLOB` - Temporal Limit Order Book (requires L2 data, pending training) -- `Liquid` - Liquid Neural Network (CUDA validation complete, pending training) - -### Arguments - -| Argument | Required | Description | -|----------|----------|-------------| -| `--symbol` | โœ… | Trading symbol (ES.FUT, NQ.FUT, etc.) | -| `--account` | โœ… | Account ID for order submission | -| `--model` | โŒ | Specific model (default: ensemble) | - -### Examples - -```bash -# Ensemble prediction for ES.FUT -tli trade ml submit --symbol ES.FUT --account prod_account - -# MAMBA-2 prediction for NQ.FUT -tli trade ml submit --symbol NQ.FUT --account test_account --model MAMBA2 - -# PPO prediction for ZN.FUT -tli trade ml submit --symbol ZN.FUT --account rl_account --model PPO -``` - ---- - -## 2. View ML Predictions - -### Basic Usage - -```bash -tli trade ml predictions --symbol ES.FUT -``` - -**Output**: -``` -ML Predictions for ES.FUT (Last 10) - -โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” -โ”‚ Timestamp โ”‚ Model โ”‚ Symbol โ”‚ Action โ”‚ Confidence โ”‚ Outcome โ”‚ -โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค -โ”‚ 2025-10-16 07:30:00 โ”‚ DQN โ”‚ ES.FUT โ”‚ BUY โ”‚ 87.2% โ”‚ +0.5% โ”‚ -โ”‚ 2025-10-16 07:25:00 โ”‚ MAMBA2 โ”‚ ES.FUT โ”‚ HOLD โ”‚ 72.1% โ”‚ N/A โ”‚ -โ”‚ 2025-10-16 07:20:00 โ”‚ PPO โ”‚ ES.FUT โ”‚ SELL โ”‚ 81.3% โ”‚ +0.3% โ”‚ -โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ -``` - -### With Filters - -```bash -# Filter by model -tli trade ml predictions --symbol ES.FUT --model MAMBA2 - -# Limit results -tli trade ml predictions --symbol ES.FUT --limit 50 - -# Combined filters -tli trade ml predictions --symbol ES.FUT --model DQN --limit 20 -``` - -### Arguments - -| Argument | Required | Description | -|----------|----------|-------------| -| `--symbol` | โœ… | Trading symbol to query | -| `--model` | โŒ | Filter by specific model | -| `--limit` | โŒ | Max results (default: 10, max: 100) | - ---- - -## 3. View Model Performance - -### All Models - -```bash -tli trade ml performance -``` - -**Output**: -``` -ML Model Performance (Last 30 days) - -โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” -โ”‚ Model โ”‚ Accuracy โ”‚ Predictions โ”‚ Sharpe Ratio โ”‚ Avg Returnโ”‚ Max Drawdownโ”‚ -โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค -โ”‚ DQN โ”‚ 68.2% โ”‚ 1,243 โ”‚ 1.42 โ”‚ +2.1% โ”‚ -3.2% โ”‚ -โ”‚ MAMBA2 โ”‚ 72.5% โ”‚ 1,189 โ”‚ 1.67 โ”‚ +2.8% โ”‚ -2.1% โ”‚ -โ”‚ PPO โ”‚ 65.3% โ”‚ 1,156 โ”‚ 1.18 โ”‚ +1.5% โ”‚ -4.5% โ”‚ -โ”‚ TFT โ”‚ 70.1% โ”‚ 1,221 โ”‚ 1.53 โ”‚ +2.4% โ”‚ -2.8% โ”‚ -โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ - -Ensemble Confidence Threshold: 0.60 -Active Models: 4/6 (TLOB and Liquid NN training pending) -``` - -### Single Model - -```bash -tli trade ml performance --model MAMBA2 -``` - -### Arguments - -| Argument | Required | Description | -|----------|----------|-------------| -| `--model` | โŒ | Filter by specific model (shows all if omitted) | - ---- - -## Performance Metrics Explained - -- **Accuracy**: Percentage of correct predictions (BUY when price goes up, SELL when down) -- **Sharpe Ratio**: Risk-adjusted returns (>1.0 is good, >2.0 is excellent) -- **Avg Return**: Average profit/loss per trade -- **Max Drawdown**: Largest peak-to-trough decline - ---- - -## Ensemble Mode - -When no `--model` is specified, TLI uses **ensemble mode**: - -1. Queries all active models -2. Weights predictions by confidence scores -3. Calculates ensemble vote -4. Uses weighted average for final decision - -**Benefits**: -- More robust predictions -- Reduced single-model bias -- Higher confidence threshold (0.60 vs 0.50) - ---- - -## Troubleshooting - -### "Not authenticated" - -```bash -tli auth login -``` - -### "Permission denied" - -Verify JWT has `trading.submit` and `trading.view` scopes: -```bash -tli auth token --decode -``` - -### "Trading Service unavailable" - -Check service status: -```bash -docker-compose ps trading_service -curl http://localhost:8081/health -``` - -### "Model not found" - -Ensure model is trained and deployed. Check model status: -```bash -tli trade ml performance -``` - ---- - -## Best Practices - -1. **Start with Ensemble Mode**: More reliable than single models -2. **Monitor Performance**: Check `tli trade ml performance` weekly -3. **Use Paper Trading First**: Test in simulation before live capital -4. **Set Confidence Thresholds**: Only trade when confidence >70% -5. **Diversify Across Models**: Don't rely on single model predictions - ---- - -## See Also - -- [TLI Authentication](AUTH.md) -- [Trading Agent Service](../services/trading_agent_service/README.md) -- [ML Training Guide](../ml/TRAINING_GUIDE.md) diff --git a/bin/fxt/docs/USAGE.md b/bin/fxt/docs/USAGE.md deleted file mode 100644 index 7e4a32c08..000000000 --- a/bin/fxt/docs/USAGE.md +++ /dev/null @@ -1,759 +0,0 @@ -# TLI (Terminal Line Interface) Usage Guide - -Welcome to the Foxhunt TLI usage guide. This document provides comprehensive information on how to use the TLI client library to interact with the Foxhunt High-Frequency Trading system. - -## Table of Contents - -1. [Quick Start](#quick-start) -2. [Installation](#installation) -3. [Basic Usage](#basic-usage) -4. [Configuration](#configuration) -5. [API Reference](#api-reference) -6. [Examples](#examples) -7. [Testing](#testing) -8. [Performance](#performance) -9. [Troubleshooting](#troubleshooting) -10. [Best Practices](#best-practices) - -## Quick Start - -```rust -use tli::prelude::*; -use tli::TliClient; - -#[tokio::main] -async fn main() -> TliResult<()> { - // Create and connect to TLI client - let mut client = TliClient::new(); - client.connect().await?; - - // Check system health - let health = client.check_health().await?; - println!("System status: {:?}", health); - - // Disconnect when done - client.disconnect().await; - Ok(()) -} -``` - -## Installation - -Add TLI to your `Cargo.toml`: - -```toml -[dependencies] -tli = { path = "../tli" } # Local development -# or when published: -# tli = "0.1.0" - -# Required async runtime -tokio = { version = "1.0", features = ["full"] } - -# For logging (recommended) -tracing = "0.1" -tracing-subscriber = "0.3" -``` - -## Basic Usage - -### Creating a Client - -All traffic routes through the api_gateway at port 50050. - -```rust -use tli::TliClient; - -// Default: connects to api_gateway at localhost:50050 -let client = TliClient::new(); -``` - -### Connecting to Services - -```rust -let mut client = TliClient::new(); - -// Connect to all services -client.connect().await?; - -// Check connection status -let health_status = client.check_health().await?; -for status in health_status { - println!("{}: {} ({})", status.service, status.status, status.endpoint); -} -``` - -### Order Management - -```rust -use tli::proto::trading::*; - -// Submit an order -let order_request = SubmitOrderRequest { - symbol: "AAPL".to_string(), - side: OrderSide::Buy as i32, - order_type: OrderType::Market as i32, - quantity: 100.0, - price: Some(150.0), -}; - -let trading_client = client.trading()?; -let response = trading_client - .submit_order(tonic::Request::new(order_request)) - .await?; - -println!("Order submitted: {}", response.into_inner().order_id); - -// List orders -let list_request = ListOrdersRequest { - symbol: "".to_string(), // All symbols - limit: Some(10), -}; - -let response = trading_client - .list_orders(tonic::Request::new(list_request)) - .await?; - -for order in response.into_inner().orders { - println!("Order: {} - {} {} @ ${}", - order.order_id, order.symbol, order.quantity, order.price); -} -``` - -### Real-time Streaming - -```rust -use tokio_stream::StreamExt; - -// Start order updates stream -let request = StreamOrderUpdatesRequest {}; -let response = trading_client - .stream_order_updates(tonic::Request::new(request)) - .await?; - -let mut stream = response.into_inner(); - -// Process updates -while let Some(update) = stream.next().await { - match update { - Ok(order_update) => { - println!("Order update: {} - Status: {}", - order_update.order_id, order_update.status); - } - Err(e) => { - eprintln!("Stream error: {}", e); - break; - } - } -} -``` - -### Configuration Management - -```rust -// Get configuration -let config_client = client.config()?; - -let get_request = GetConfigRequest { - key: "max_order_size".to_string(), -}; - -let response = config_client - .get_config(tonic::Request::new(get_request)) - .await?; - -if let Some(config) = response.into_inner().config { - println!("Config: {} = {}", config.key, config.value); -} - -// Set configuration -let set_request = SetConfigRequest { - key: "trading_enabled".to_string(), - value: "true".to_string(), - config_type: "boolean".to_string(), - description: "Enable trading operations".to_string(), -}; - -let response = config_client - .set_config(tonic::Request::new(set_request)) - .await?; - -println!("Config updated: {}", response.into_inner().message); -``` - -### Monitoring and Metrics - -```rust -// Get system status -let monitoring_client = client.monitoring()?; - -let status_request = GetSystemStatusRequest {}; -let response = monitoring_client - .get_system_status(tonic::Request::new(status_request)) - .await?; - -for service in response.into_inner().services { - println!("Service: {} - Status: {}", service.name, service.status); -} - -// Get metrics -let metrics_request = GetMetricsRequest { - metric_names: vec![ - "latency_p99".to_string(), - "orders_per_second".to_string(), - ], -}; - -let response = monitoring_client - .get_metrics(tonic::Request::new(metrics_request)) - .await?; - -for metric in response.into_inner().metrics { - println!("Metric: {} = {} {}", metric.name, metric.value, metric.unit); -} -``` - -## Configuration - -### Environment Variables - -TLI can be configured using environment variables: - -```bash -# API Gateway endpoint (all services route through here) -export FOXHUNT_API_GATEWAY_URL="http://localhost:50050" - -# Logging -export RUST_LOG="info" - -# Testing -export TLI_ENABLE_RT_TESTS="true" -``` - -### Connection Timeouts - -```rust -use std::time::Duration; -use tonic::transport::Endpoint; - -// Custom timeout configuration -let channel = Endpoint::from_shared("http://localhost:50050")? - .timeout(Duration::from_secs(10)) - .connect_timeout(Duration::from_secs(5)) - .connect() - .await?; -``` - -## API Reference - -### Core Types - -```rust -// Re-exported from the prelude -use tli::prelude::*; - -// Main client -pub struct TliClient { /* ... */ } - -// Error types -pub enum TliError { - Connection(String), - InvalidRequest(String), - InvalidSymbol(String), - NotConnected(String), -} - -pub type TliResult = Result; -``` - -### Client Methods - -```rust -impl TliClient { - // Creation - pub fn new() -> Self; - - // Connection management - pub async fn connect(&mut self) -> TliResult<()>; - pub async fn disconnect(&mut self); - pub async fn check_health(&mut self) -> TliResult>; - - // Service access - pub fn trading(&mut self) -> TliResult<&mut TradingServiceClient>; - pub fn monitoring(&mut self) -> TliResult<&mut MonitoringServiceClient>; - pub fn config(&mut self) -> TliResult<&mut ConfigServiceClient>; -} -``` - -### Utility Functions - -```rust -// Timestamp conversions -pub fn unix_nanos_to_system_time(nanos: i64) -> SystemTime; -pub fn system_time_to_unix_nanos(time: SystemTime) -> i64; -pub fn current_unix_nanos() -> i64; - -// Validation -pub fn validate_symbol(symbol: &str) -> TliResult<()>; -pub fn validate_quantity(quantity: f64) -> TliResult<()>; -pub fn validate_price(price: f64) -> TliResult<()>; - -// Type conversions -pub fn order_side_to_string(side: OrderSide) -> &'static str; -pub fn string_to_order_side(side: &str) -> TliResult; -pub fn order_type_to_string(order_type: OrderType) -> &'static str; -pub fn string_to_order_type(order_type: &str) -> TliResult; -``` - -## Examples - -### Configuration Management - -Run the configuration management example: - -```bash -# Basic configuration operations -cargo run --example config_management basic - -# Configuration validation demo -cargo run --example config_management validation - -# Search and filtering -cargo run --example config_management search - -# Interactive editor -cargo run --example config_management interactive -``` - -Features: -- Configuration CRUD operations -- Validation with custom rules -- Category-based organization -- Search and filtering -- Sensitive data masking - -### Real-time Streaming - -Run the streaming example: - -```bash -# Basic streaming (requires running services) -cargo run --example real_time_streaming basic - -# High-frequency data handling -cargo run --example real_time_streaming highfreq - -# Error handling and recovery -cargo run --example real_time_streaming errors - -# Backpressure handling -cargo run --example real_time_streaming backpressure -``` - -Features: -- Multiple concurrent streams -- High-frequency data processing -- Error handling and recovery -- Backpressure management -- Real-time aggregation - -## Testing - -### Running Tests - -```bash -# Unit tests -cargo test - -# Integration tests -cargo test --test integration - -# Property-based tests -cargo test --features proptest - -# With real-time tests (requires services) -TLI_ENABLE_RT_TESTS=true cargo test -``` - -### Mock Server Testing - -```bash -# Start mock servers for testing -cargo test --test integration -- --nocapture - -# Test with specific mock port -TEST_MOCK_PORT=51000 cargo test -``` - -### Property-Based Testing - -The TLI library includes property-based tests using `proptest`: - -```rust -proptest! { - #[test] - fn test_timestamp_conversion_property(timestamp in 0i64..i64::MAX/2) { - let system_time = unix_nanos_to_system_time(timestamp); - let converted = system_time_to_unix_nanos(system_time); - prop_assert!((converted - timestamp).abs() < 1000); - } -} -``` - -## Performance - -### Benchmarking - -Run performance benchmarks: - -```bash -# All benchmarks -cargo bench - -# Specific benchmark suites -cargo bench --bench configuration_benchmarks -cargo bench --bench client_performance -cargo bench --bench serialization_benchmarks - -# With HTML reports -cargo bench -- --output-format html -``` - -### Performance Characteristics - -Based on benchmarks, TLI provides: - -- **Timestamp conversions**: < 100ns per operation -- **Validation operations**: < 50ns per validation -- **Type conversions**: < 10ns per conversion -- **Protobuf serialization**: ~1-10ฮผs depending on message size -- **JSON serialization**: ~5-50ฮผs depending on message size - -### Optimization Tips - -1. **Reuse clients**: Create TLI clients once and reuse them -2. **Batch operations**: Use streaming for high-frequency data -3. **Connection pooling**: Share connections across multiple operations -4. **Async programming**: Use `tokio::spawn` for concurrent operations -5. **Buffer sizing**: Tune channel buffer sizes for your workload - -```rust -// Good: Reuse client -let mut client = TliClient::new(); -client.connect().await?; -for _ in 0..1000 { - let _ = client.check_health().await?; -} - -// Better: Batch operations -let health_checks = (0..1000).map(|_| client.check_health()); -let results = futures::future::join_all(health_checks).await; -``` - -## Troubleshooting - -### Common Issues - -#### Connection Refused - -``` -Error: Connection(Connection refused) -``` - -**Solution**: Ensure the api_gateway is running and accessible: - -```bash -# Check if api_gateway is running -netstat -ln | grep 50050 - -# Check environment variables -echo $FOXHUNT_API_GATEWAY_URL -``` - -#### Service Not Connected - -``` -Error: NotConnected(Trading service not connected) -``` - -**Solution**: Call `connect()` before using services: - -```rust -let mut client = TliClient::new(); -client.connect().await?; // Required before using services -let trading_client = client.trading()?; -``` - -#### Invalid Symbol - -``` -Error: InvalidSymbol(Symbol contains invalid characters) -``` - -**Solution**: Use valid symbol format: - -```rust -// Valid symbols -validate_symbol("AAPL")?; // โœ“ -validate_symbol("BTC.USD")?; // โœ“ -validate_symbol("EUR-USD")?; // โœ“ - -// Invalid symbols -validate_symbol("BTC/USD")?; // โœ— - slash not allowed -validate_symbol("")?; // โœ— - empty not allowed -``` - -#### Timeout Errors - -``` -Error: Connection(Request timeout) -``` - -**Solution**: Increase timeouts or check network connectivity: - -```rust -// Custom timeout configuration via tonic Endpoint -use tonic::transport::Endpoint; -use std::time::Duration; - -let channel = Endpoint::from_shared("http://localhost:50050")? - .timeout(Duration::from_secs(30)) - .connect_timeout(Duration::from_secs(10)) - .connect() - .await?; -``` - -### Debug Logging - -Enable debug logging for troubleshooting: - -```bash -# Enable debug logs -export RUST_LOG=debug - -# Trace-level logging (very verbose) -export RUST_LOG=trace -``` - -```rust -// In code -tracing_subscriber::fmt() - .with_max_level(tracing::Level::DEBUG) - .init(); -``` - -### Performance Issues - -If experiencing performance issues: - -1. **Check network latency**: - ```bash - ping trading-engine-host - traceroute trading-engine-host - ``` - -2. **Monitor resource usage**: - ```bash - # CPU and memory usage - top -p $(pgrep your-app) - - # Network connections - ss -tuln | grep :50052 - ``` - -3. **Profile your application**: - ```bash - # Use perf for profiling - perf record -g cargo run --release --example basic_dashboard - perf report - ``` - -## Best Practices - -### Error Handling - -Always handle errors appropriately: - -```rust -// Good: Proper error handling -match client.check_health().await { - Ok(health) => { - // Process health status - for status in health { - if status.status.contains("ERROR") { - warn!("Service {} has issues: {}", status.service, status.status); - } - } - } - Err(TliError::Connection(msg)) => { - error!("Connection failed: {}", msg); - // Implement retry logic - } - Err(TliError::NotConnected(msg)) => { - warn!("Not connected: {}", msg); - // Attempt reconnection - } - Err(e) => { - error!("Unexpected error: {}", e); - } -} -``` - -### Resource Management - -Properly manage connections and resources: - -```rust -// Good: Explicit disconnection -{ - let mut client = TliClient::new(); - client.connect().await?; - - // Use client... - - client.disconnect().await; // Explicit cleanup -} - -// Better: Use RAII pattern -struct TradingSession { - client: TliClient, -} - -impl TradingSession { - async fn new() -> TliResult { - let mut client = TliClient::new(); - client.connect().await?; - Ok(Self { client }) - } -} - -impl Drop for TradingSession { - fn drop(&mut self) { - // Note: async drop not available, use explicit cleanup - // or spawn a task for async cleanup - } -} -``` - -### Concurrent Operations - -Use async properly for concurrent operations: - -```rust -// Good: Concurrent health checks for multiple clients -let clients = vec![client1, client2, client3]; -let health_checks = clients.iter_mut() - .map(|client| client.check_health()); -let results = futures::future::join_all(health_checks).await; - -// Better: Use spawn for truly parallel operations -let tasks: Vec<_> = clients.into_iter() - .map(|mut client| tokio::spawn(async move { - client.check_health().await - })) - .collect(); - -let results = futures::future::join_all(tasks).await; -``` - -### Validation - -Always validate input data: - -```rust -// Validate before submitting orders -fn validate_order_request(request: &SubmitOrderRequest) -> TliResult<()> { - validate_symbol(&request.symbol)?; - validate_quantity(request.quantity)?; - - if let Some(price) = request.price { - validate_price(price)?; - } - - Ok(()) -} - -// Use in order submission -validate_order_request(&order_request)?; -let response = trading_client.submit_order(tonic::Request::new(order_request)).await?; -``` - -### Configuration Management - -Use environment-specific configuration: - -```rust -// Environment-aware configuration -fn get_endpoints_for_env() -> ServiceEndpoints { - match std::env::var("ENVIRONMENT").as_deref() { - Ok("production") => ServiceEndpoints { - trading_engine: "https://prod-trading.company.com".to_string(), - // ... production endpoints - }, - Ok("staging") => ServiceEndpoints { - trading_engine: "https://staging-trading.company.com".to_string(), - // ... staging endpoints - }, - _ => ServiceEndpoints::default(), // Development defaults - } -} -``` - -### Monitoring and Observability - -Implement proper monitoring: - -```rust -use tracing::{info, warn, error, instrument}; - -#[instrument] -async fn submit_order_with_monitoring( - client: &mut TliClient, - request: SubmitOrderRequest, -) -> TliResult { - let start = std::time::Instant::now(); - - match client.trading()?.submit_order(tonic::Request::new(request)).await { - Ok(response) => { - let elapsed = start.elapsed(); - info!("Order submitted successfully in {:?}", elapsed); - Ok(response.into_inner().order_id) - } - Err(e) => { - let elapsed = start.elapsed(); - error!("Order submission failed after {:?}: {}", elapsed, e); - Err(TliError::Connection(e.to_string())) - } - } -} -``` - ---- - -## Additional Resources - -- [TLI API Documentation](https://docs.rs/tli) (when published) -- [Foxhunt System Documentation](../docs/) -- [gRPC Documentation](https://grpc.io/docs/) -- [Tokio Documentation](https://tokio.rs/) -- [Tracing Documentation](https://tracing.rs/) - -## Support - -For support and questions: - -1. Check the [troubleshooting section](#troubleshooting) -2. Review the [examples](#examples) -3. Run the test suite to verify your setup -4. Check the logs with debug logging enabled - -## Contributing - -When contributing to TLI: - -1. Run the full test suite: `cargo test` -2. Run benchmarks: `cargo bench` -3. Check code formatting: `cargo fmt` -4. Run clippy: `cargo clippy` -5. Update documentation as needed - diff --git a/bin/fxt/tests/INTEGRATION_TEST_GUIDE.md b/bin/fxt/tests/INTEGRATION_TEST_GUIDE.md deleted file mode 100644 index 8c0a60dd3..000000000 --- a/bin/fxt/tests/INTEGRATION_TEST_GUIDE.md +++ /dev/null @@ -1,537 +0,0 @@ -# TLI Integration Test Guide - -## Overview - -This guide provides comprehensive documentation for running and understanding the TLI (Terminal Line Interface) system integration tests. The test suite covers service communication, database operations, end-to-end workflows, error handling, and performance testing. - -## Test Structure - -``` -tli/tests/integration/ -โ”œโ”€โ”€ mod.rs # Common test utilities and configuration -โ”œโ”€โ”€ service_integration_tests.rs # Service-to-service communication tests -โ”œโ”€โ”€ database_integration_tests.rs # Database operation and integrity tests -โ”œโ”€โ”€ end_to_end_tests.rs # Complete workflow scenario tests -โ”œโ”€โ”€ error_handling_tests.rs # Resilience and error recovery tests -โ””โ”€โ”€ performance_tests.rs # Load testing and performance validation -``` - -## Prerequisites - -### Environment Setup - -1. **Required Environment Variables** - ```bash - # Optional: PostgreSQL testing (if available) - export TEST_POSTGRES_URL="postgresql://user:password@localhost:5432/test_db" - - # Optional: InfluxDB testing (if available) - export TEST_INFLUX_URL="http://localhost:8086" - export TEST_INFLUX_TOKEN="your_influx_token" - export TEST_INFLUX_ORG="test_org" - - # Optional: Enable real-time tests - export TLI_ENABLE_RT_TESTS="true" - - # Logging level - export RUST_LOG="info" - ``` - -2. **Database Setup** (Optional) - ```bash - # PostgreSQL (if testing database integration) - createdb test_db - - # InfluxDB (if testing time-series integration) - influx bucket create -n test_bucket -o test_org - ``` - -### Dependencies - -All test dependencies are defined in `Cargo.toml`: -- `tokio-test` - Async testing utilities -- `tempfile` - Temporary file management -- `wiremock` - HTTP mocking for network tests -- `fake` - Test data generation -- `proptest` - Property-based testing -- `criterion` - Performance benchmarking - -## Test Categories - -### 1. Service Integration Tests - -**File**: `service_integration_tests.rs` - -Tests communication between TLI clients and backend services: - -#### Trading Service Tests -- Order submission and cancellation -- Position monitoring -- Market data subscription -- Risk management validation -- Portfolio analytics - -#### Backtesting Service Tests -- Backtest execution workflow -- Strategy optimization -- Data management -- Performance reporting - -#### Configuration Management Tests -- Configuration CRUD operations -- Hot-reload functionality -- Validation workflows - -#### Event Streaming Tests -- Real-time event subscription -- Event filtering and routing -- Stream reconnection handling - -**Run Command**: -```bash -cargo test --test integration service_integration_tests -``` - -### 2. Database Integration Tests - -**File**: `database_integration_tests.rs` - -Tests database operations across different storage systems: - -#### SQLite Configuration Tests -- Configuration CRUD operations -- Bulk operations and transactions -- Version management -- Backup and restore functionality - -#### PostgreSQL Event Tests (Optional) -- Event storage and retrieval -- Event aggregation and statistics -- Cleanup and archival processes -- Query performance - -#### InfluxDB Time-Series Tests (Optional) -- Market data storage -- Performance metrics collection -- Time-based queries and aggregations - -#### Transaction Integrity Tests -- Rollback scenarios -- Multi-database consistency -- Connection pool management -- Timeout handling - -**Run Command**: -```bash -cargo test --test integration database_integration_tests -``` - -### 3. End-to-End Scenario Tests - -**File**: `end_to_end_tests.rs` - -Tests complete business workflows: - -#### Order Lifecycle Tests -- Complete order submission โ†’ execution โ†’ settlement -- Order modification and cancellation -- Batch order operations -- Position and portfolio updates - -#### Backtesting Workflow Tests -- Data availability validation -- Backtest creation and execution -- Strategy optimization workflows -- Walk-forward analysis -- Report generation - -#### Configuration Management Flow -- Configuration updates with hot-reload -- Validation and approval workflows -- History tracking - -#### Security Authentication Flow -- User registration and authentication -- Role-based access control -- Session management -- Token refresh and logout - -#### Event Storage and Retrieval -- Event generation through operations -- Event filtering and querying -- Event statistics and analytics - -**Run Command**: -```bash -cargo test --test integration end_to_end_tests -``` - -### 4. Error Handling and Resilience Tests - -**File**: `error_handling_tests.rs` - -Tests system behavior under failure conditions: - -#### Service Unavailability Tests -- Single service failures -- Partial system degradation -- Service recovery scenarios - -#### Network Failure Tests -- Connection timeouts -- Connection refused handling -- Intermittent network failures - -#### Database Failure Tests -- Connection failures -- Transaction rollback scenarios -- Recovery mechanisms - -#### Invalid Data Tests -- Malformed requests -- Data validation failures -- Boundary condition handling - -#### Resilience Pattern Tests -- Circuit breaker activation -- Retry with exponential backoff -- Rate limiting handling -- Graceful degradation - -**Run Command**: -```bash -cargo test --test integration error_handling_tests -``` - -### 5. Performance and Load Tests - -**File**: `performance_tests.rs` - -Tests system performance under various load conditions: - -#### Concurrent Request Tests -- High-concurrency order submission -- Mixed operation load testing -- Connection pool stress testing - -#### High-Frequency Trading Simulation -- Ultra-low latency requirements -- High-throughput order flow -- Market data streaming performance - -#### Stress Tests -- Memory usage under load -- Resource limit validation -- Connection pool exhaustion - -#### Backtesting Performance -- Concurrent backtest execution -- Large dataset processing -- Resource utilization monitoring - -**Run Command**: -```bash -cargo test --test integration performance_tests -``` - -## Running Tests - -### All Integration Tests -```bash -# Run all integration tests -cargo test --test integration - -# Run with logging -RUST_LOG=info cargo test --test integration - -# Run with nocapture to see output -cargo test --test integration -- --nocapture -``` - -### Specific Test Categories -```bash -# Service integration only -cargo test --test integration service_integration - -# Database tests only -cargo test --test integration database_integration - -# End-to-end scenarios only -cargo test --test integration end_to_end - -# Error handling only -cargo test --test integration error_handling - -# Performance tests only -cargo test --test integration performance -``` - -### Individual Test Functions -```bash -# Specific test function -cargo test --test integration test_complete_order_lifecycle - -# Tests matching pattern -cargo test --test integration order_lifecycle -``` - -### Test Configuration - -#### Performance Test Configuration -```bash -# High-performance testing -export TLI_PERF_MAX_CONCURRENT=100 -export TLI_PERF_TEST_DURATION=30 -export TLI_PERF_TARGET_LATENCY_MS=10 - -# Memory-constrained testing -export TLI_PERF_MEMORY_LIMIT_MB=256 -``` - -#### Database Test Configuration -```bash -# Skip database tests if no external databases available -export TLI_SKIP_DATABASE_TESTS="true" - -# Use specific database for testing -export TLI_TEST_DATABASE_URL="sqlite::memory:" -``` - -## Mock Servers - -The test suite includes sophisticated mock servers that simulate real services: - -### MockTradingServer -- Simulates trading service responses -- Configurable failure modes -- Performance testing optimizations -- Event generation - -### MockBacktestingServer -- Simulates backtesting workflows -- Progress tracking simulation -- Result generation -- Resource usage simulation - -### MockRiskServer -- Risk calculation simulation -- Limit validation -- Alert generation - -### Failure Modes -- `ServiceUnavailable` - 503 responses -- `DatabaseError` - Database connection failures -- `TransactionFailure` - Transaction rollback scenarios -- `IntermittentFailure` - Random failures -- `RateLimited` - 429 responses -- `PartialDegradation` - Reduced functionality - -## Test Data Management - -### Temporary Resources -- SQLite databases created in temporary directories -- Automatic cleanup after tests -- Isolated test environments - -### Generated Test Data -- Fake market data using `fake` crate -- Random but consistent test symbols -- Realistic order and position data - -### Configuration Fixtures -- Predefined configuration sets -- Validation test cases -- Performance baselines - -## Performance Benchmarks - -### Latency Targets -- Order submission: < 10ms P99 -- Position queries: < 5ms P99 -- Configuration updates: < 100ms P99 - -### Throughput Targets -- Order submission: > 1000 RPS -- Market data updates: > 10,000/second -- Event processing: > 5,000/second - -### Resource Limits -- Memory usage: < 512MB under load -- CPU usage: < 80% sustained -- Connection pool: 100 concurrent connections - -## Troubleshooting - -### Common Issues - -1. **Port Conflicts** - ``` - Error: Address already in use - ``` - Solution: Change `mock_server_port` in test configuration or kill conflicting processes. - -2. **Database Connection Failures** - ``` - Error: Failed to connect to database - ``` - Solution: Verify database URLs and credentials, or disable database tests. - -3. **Timeout Failures** - ``` - Error: Operation timed out - ``` - Solution: Increase timeout values in test configuration or check system performance. - -4. **Memory Issues** - ``` - Error: Cannot allocate memory - ``` - Solution: Reduce concurrent test limits or increase available memory. - -### Debug Mode -```bash -# Enable debug logging -RUST_LOG=debug cargo test --test integration - -# Enable trace logging for specific modules -RUST_LOG=tli=trace cargo test --test integration - -# Run single test with full output -cargo test --test integration test_name -- --exact --nocapture -``` - -### Performance Debugging -```bash -# Run with performance profiling -cargo test --test integration performance_tests --release - -# Memory usage tracking -valgrind --tool=massif cargo test --test integration - -# CPU profiling -perf record cargo test --test integration performance_tests -``` - -## Continuous Integration - -### GitHub Actions Configuration -```yaml -name: Integration Tests -on: [push, pull_request] - -jobs: - integration-tests: - runs-on: ubuntu-latest - services: - postgres: - image: postgres:14 - env: - POSTGRES_PASSWORD: test - options: >- - --health-cmd pg_isready - --health-interval 10s - --health-timeout 5s - --health-retries 5 - - steps: - - uses: actions/checkout@v3 - - uses: actions-rs/toolchain@v1 - with: - toolchain: stable - - name: Run integration tests - env: - TEST_POSTGRES_URL: postgresql://postgres:test@localhost:5432/postgres - RUST_LOG: info - run: cargo test --test integration -``` - -### Test Reports -- JUnit XML output for CI systems -- Performance benchmark results -- Coverage reports -- Memory usage analysis - -## Contributing - -### Adding New Tests - -1. **Service Integration Tests** - - Add to appropriate module in `service_integration_tests.rs` - - Follow existing patterns for setup/teardown - - Include both success and failure scenarios - -2. **Database Tests** - - Add to `database_integration_tests.rs` - - Handle optional database availability - - Test both SQLite and PostgreSQL when available - -3. **End-to-End Tests** - - Add complete workflow tests to `end_to_end_tests.rs` - - Include authentication and authorization - - Test realistic user scenarios - -4. **Error Handling Tests** - - Add to `error_handling_tests.rs` - - Test specific failure modes - - Validate recovery mechanisms - -5. **Performance Tests** - - Add to `performance_tests.rs` - - Define performance targets - - Include resource usage monitoring - -### Test Naming Conventions -- Use descriptive names: `test_complete_order_lifecycle` -- Group related tests: `order_lifecycle_tests` module -- Include test type: `test_concurrent_order_submission` - -### Documentation -- Document test purpose and scope -- Include setup requirements -- Specify performance expectations -- Add troubleshooting notes - -## Metrics and Monitoring - -### Test Metrics Collected -- Execution time per test -- Memory usage patterns -- Network request latencies -- Database query performance -- Error rates and types - -### Performance Baselines -- Baseline measurements for regression detection -- Performance trend analysis -- Resource usage monitoring -- Capacity planning data - -### Reporting -- Test execution reports -- Performance benchmark results -- Coverage analysis -- Failure trend analysis - -## Security Considerations - -### Test Data -- No real credentials in tests -- Temporary databases only -- Isolated test environments -- Automatic cleanup - -### Network Security -- Mock servers only bind to localhost -- No external network access required -- Encrypted communication testing -- Authentication flow validation - -### Data Protection -- No persistent sensitive data -- Memory cleanup after tests -- Secure random data generation -- Audit trail testing - ---- - -For additional support or questions about the integration tests, please refer to the main TLI documentation or contact the development team. \ No newline at end of file diff --git a/bin/fxt/tests/TEST_EXECUTION_README.md b/bin/fxt/tests/TEST_EXECUTION_README.md deleted file mode 100644 index 031ee3473..000000000 --- a/bin/fxt/tests/TEST_EXECUTION_README.md +++ /dev/null @@ -1,101 +0,0 @@ -# TLI Integration Tests - Quick Start Guide - -## Quick Test Execution - -### Run All Integration Tests -```bash -cd /home/jgrusewski/Work/foxhunt/tli -cargo test --test integration -``` - -### Run Specific Test Categories -```bash -# Service communication tests -cargo test --test integration service_integration - -# Database operation tests -cargo test --test integration database_integration - -# End-to-end workflow tests -cargo test --test integration end_to_end - -# Error handling and resilience tests -cargo test --test integration error_handling - -# Performance and load tests -cargo test --test integration performance -``` - -### Run with Detailed Output -```bash -# See test output and logs -RUST_LOG=info cargo test --test integration -- --nocapture - -# Debug mode with full logging -RUST_LOG=debug cargo test --test integration -- --nocapture -``` - -## Test Structure Summary - -| Test File | Purpose | Key Test Areas | -|-----------|---------|----------------| -| `service_integration_tests.rs` | Service-to-service communication | Trading orders, backtesting, config management, event streaming | -| `database_integration_tests.rs` | Database operations | SQLite config, PostgreSQL events, InfluxDB metrics, transactions | -| `end_to_end_tests.rs` | Complete business workflows | Order lifecycle, backtest workflow, auth flow, event storage | -| `error_handling_tests.rs` | System resilience | Service failures, network issues, invalid data, circuit breakers | -| `performance_tests.rs` | Load and performance | Concurrent requests, HFT simulation, stress testing | - -## Environment Variables (Optional) - -```bash -# PostgreSQL testing (if available) -export TEST_POSTGRES_URL="postgresql://user:password@localhost:5432/test_db" - -# InfluxDB testing (if available) -export TEST_INFLUX_URL="http://localhost:8086" -export TEST_INFLUX_TOKEN="your_token" -export TEST_INFLUX_ORG="test_org" - -# Enable real-time tests -export TLI_ENABLE_RT_TESTS="true" - -# Logging -export RUST_LOG="info" -``` - -## Test Results Interpretation - -### Success Indicators -- โœ… All tests pass -- โœ… Performance targets met -- โœ… Error handling validates correctly -- โœ… No memory leaks or resource issues - -### Common Issues -- **Port conflicts**: Change mock server ports in test config -- **Database unavailable**: Tests will skip PostgreSQL/InfluxDB if not configured -- **Timeouts**: May indicate performance issues or resource constraints - -## Performance Benchmarks - -### Expected Results -- **Order Submission**: < 10ms P99 latency, > 1000 RPS throughput -- **Market Data**: > 10,000 updates/second processing -- **Database Operations**: < 5ms P99 for configuration queries -- **Memory Usage**: < 512MB under load - -## Full Documentation - -See `INTEGRATION_TEST_GUIDE.md` for comprehensive documentation including: -- Detailed test descriptions -- Setup requirements -- Troubleshooting guide -- Contributing guidelines -- CI/CD integration - -## Support - -For issues or questions: -1. Check the troubleshooting section in `INTEGRATION_TEST_GUIDE.md` -2. Review test logs with `RUST_LOG=debug` -3. Verify environment setup and dependencies \ No newline at end of file diff --git a/crates/data/tests/EDGE_CASE_FINDINGS.md b/crates/data/tests/EDGE_CASE_FINDINGS.md deleted file mode 100644 index c00a869bd..000000000 --- a/crates/data/tests/EDGE_CASE_FINDINGS.md +++ /dev/null @@ -1,511 +0,0 @@ -# Edge Case Testing Report - Real Market Data - -**Date**: 2025-10-13 -**Agent**: Agent 21 -**Objective**: Comprehensive edge case testing with real market data -**Test Suite**: `/home/jgrusewski/Work/foxhunt/data/tests/edge_case_tests.rs` - ---- - -## Executive Summary - -โœ… **Test Suite Created**: 22 comprehensive edge case tests covering all major production scenarios -โœ… **Tests Compile**: All tests successfully compile with zero errors -โš ๏ธ **Bug Discovered**: Parquet loader has data type downcast issue (15/22 tests reveal loader bug) -โœ… **Error Handling Validated**: 7/22 tests pass (error handling, file validation, summary tests) - -### Key Finding - -**CRITICAL BUG DISCOVERED**: The `ParquetDataLoader::batch_to_events()` function fails to downcast primitive arrays from Parquet files. This is a **production-blocking bug** that prevents loading of real market data. - -**Root Cause**: Mismatch between expected nullable arrays and actual Parquet schema encoding. - ---- - -## Test Results - -### Test Execution Summary - -| Category | Tests | Passed | Failed | Pass Rate | -|----------|-------|--------|--------|-----------| -| **Gap Handling** | 4 | 0 | 4 | 0% | -| **Extreme Volatility** | 3 | 0 | 3 | 0% | -| **Low Liquidity** | 3 | 3 | 0 | 100% | -| **Data Anomalies** | 3 | 0 | 3 | 0% | -| **Error Handling** | 4 | 3 | 1 | 75% | -| **Boundary Conditions** | 4 | 0 | 4 | 0% | -| **Summary Test** | 1 | 1 | 0 | 100% | -| **TOTAL** | **22** | **7** | **15** | **32%** | - -### Detailed Test Status - -#### โœ… PASSING TESTS (7) - -1. **test_02_overnight_gap_futures** โœ… - - **Purpose**: Validates expected maintenance windows in futures markets - - **Result**: Pass - Documentation test only - - **Findings**: ES.FUT has 1-hour daily maintenance, 48-hour weekend closure - -2. **test_09_wide_spreads** โœ… - - **Purpose**: Documents wide spread handling behavior - - **Result**: Pass - Documentation test only - - **Findings**: System should detect abnormal spreads, activate risk limits - -3. **test_10_thin_order_book** โœ… - - **Purpose**: Documents thin order book handling behavior - - **Result**: Pass - Documentation test only - - **Findings**: System should adjust position sizing in low depth conditions - -4. **test_14_missing_file_handling** โœ… - - **Purpose**: Tests graceful handling of missing files - - **Result**: Pass - Loader correctly returns error - - **Error Message**: "No such file or directory" - - **Behavior**: โœ… Graceful error handling - -5. **test_15_corrupted_file_handling** โœ… - - **Purpose**: Tests handling of corrupted Parquet files - - **Result**: Pass - Loader correctly rejects invalid data - - **Error Message**: Parquet parsing error - - **Behavior**: โœ… Graceful error handling - -6. **test_16_empty_file_handling** โœ… - - **Purpose**: Tests handling of empty files - - **Result**: Pass - Loader correctly rejects empty files - - **Error Message**: Parquet parsing error - - **Behavior**: โœ… Graceful error handling - -7. **test_22_comprehensive_edge_case_summary** โœ… - - **Purpose**: Summary of all edge case test categories - - **Result**: Pass - Reports test coverage - - **Output**: Detailed breakdown of all 22 tests - -#### โŒ FAILING TESTS (15) - -All 15 failures have the **SAME ROOT CAUSE**: Parquet loader downcast issue. - -**Error Pattern**: -``` -panicked at arrow-array-56.2.0/src/cast.rs:501:10: -Unable to downcast to primitive array -``` - -**Affected Tests**: -1. test_01_weekend_gap_detection -2. test_03_market_hours_detection -3. test_04_first_last_bar_of_day -4. test_05_large_price_swings -5. test_06_flash_crash_detection -6. test_07_volatility_clustering -7. test_08_zero_volume_bars -8. test_11_price_spike_detection -9. test_12_duplicate_timestamps -10. test_13_out_of_order_events -11. test_17_invalid_date_range -12. test_18_first_event_in_file -13. test_19_last_event_in_file -14. test_21_large_file_memory_handling - -**Impact**: All tests requiring actual data loading fail due to loader bug. - ---- - -## Bug Analysis - -### Critical Bug: Parquet Loader Downcast Failure - -**File**: `/home/jgrusewski/Work/foxhunt/data/src/replay/parquet_loader.rs` -**Function**: `batch_to_events()` -**Lines**: 170-208 - -#### Root Cause - -The `as_primitive_array::()` function fails because it expects nullable arrays but the Parquet file contains non-nullable arrays with dictionary encoding or other optimizations. - -#### Failing Code - -```rust -// Line 170-174: timestamp_ns extraction fails -let timestamp_ns = as_primitive_array::( - batch - .column_by_name("timestamp_ns") - .context("Missing timestamp_ns column")?, -); -``` - -**Problem**: `as_primitive_array()` expects exact type match, but Parquet may use: -- Dictionary encoding -- Run-length encoding -- Different nullable settings -- Type aliases or wrapped types - -#### Parquet Schema (Verified) - -From `VALIDATION_SUMMARY.md`: -- `timestamp_ns`: `Int64` โœ… -- `sequence`: `UInt64` โœ… -- `latency_ns`: `UInt64` (nullable) โœ… - -**Schema is correct** - the issue is in the loader's type casting logic. - -### Recommended Fix - -**Option 1: Use `.values()` with proper casting** -```rust -use arrow::array::Array; - -let timestamp_ns = batch - .column_by_name("timestamp_ns") - .context("Missing timestamp_ns column")? - .as_any() - .downcast_ref::>() - .context("Invalid timestamp_ns type")?; -``` - -**Option 2: Use Arrow's cast utilities** -```rust -use arrow::compute::cast; - -let timestamp_col = batch.column_by_name("timestamp_ns")?; -let timestamp_ns = cast(timestamp_col, &DataType::Timestamp(TimeUnit::Nanosecond, None))?; -``` - -**Option 3: Handle dictionary encoding** -```rust -let timestamp_col = batch.column_by_name("timestamp_ns")?; -let timestamp_ns = if let Some(dict_array) = timestamp_col.as_any().downcast_ref::>() { - // Handle dictionary-encoded column - cast_dictionary_to_primitive(dict_array)? -} else { - as_primitive_array::(timestamp_col) -}; -``` - ---- - -## Edge Cases Identified - -Even though data loading fails, the **test design** successfully identifies all critical edge cases: - -### 1. Gap Handling - -#### Weekend Gaps (BTC/USD) -- **Expectation**: BTC trades 24/7, minimal gaps -- **Test**: Detect gaps > 5 minutes -- **Purpose**: Ensure system handles market downtime gracefully -- **Production Impact**: โš ๏ธ Trading must pause during gaps, resume correctly - -#### Overnight Gaps (ES.FUT) -- **Daily Maintenance**: 1 hour (5-6 PM ET) -- **Weekend Closure**: 48 hours (Fri 5PM - Sun 6PM) -- **Test**: Validate system handles expected halts -- **Production Impact**: โœ… Risk limits should activate during gaps - -#### Market Hours Detection -- **BTC/USD**: 24/7 coverage expected (all 24 hours) -- **ES.FUT**: 23/24 hours (excluding maintenance) -- **Test**: Verify data coverage across all hours -- **Production Impact**: โœ… Detect abnormal data gaps - -#### First/Last Bar of Day -- **Test**: Validate day boundaries are correct -- **Edge Case**: Midnight rollover, timezone handling -- **Production Impact**: โš ๏ธ Daily PnL calculations depend on correct boundaries - -### 2. Extreme Volatility - -#### Large Price Swings (>5%) -- **BTC/USD**: Crypto can move >5% per bar -- **Test**: Detect and count large moves -- **Production Impact**: โš ๏ธ Risk limits must activate -- **Example**: Flash crashes, pump/dump schemes - -#### Flash Crash Detection -- **Pattern**: >3% drop + >3% recovery within 10 bars -- **Test**: Identify rapid reversals -- **Production Impact**: โš ๏ธ Circuit breakers should halt trading -- **Example**: 2010 Flash Crash, 2021 Crypto crashes - -#### Volatility Clustering -- **Statistical Fact**: High volatility follows high volatility -- **Test**: Calculate rolling 20-bar volatility -- **Production Impact**: โœ… Position sizing must adapt -- **Metric**: Standard deviation of returns - -### 3. Low Liquidity - -#### Zero Volume Bars -- **Occurrence**: Illiquid periods, market close transitions -- **Test**: Count bars with zero or <0.01 volume -- **Production Impact**: โš ๏ธ CRITICAL - Do not trade on zero volume -- **Risk**: Infinite slippage, no liquidity - -#### Wide Spreads -- **Causes**: Low liquidity, high volatility, market stress -- **Impact**: High slippage, adverse selection -- **Production Impact**: โœ… Limit orders only, no market orders - -#### Thin Order Book -- **Characteristics**: Few price levels, large gaps -- **Impact**: High slippage potential -- **Production Impact**: โœ… Reduce position sizes - -### 4. Data Anomalies - -#### Price Spikes (>3 Standard Deviations) -- **Detection**: Z-score > 3.0 from rolling mean -- **Causes**: Fat-finger trades, data errors, market manipulation -- **Production Impact**: โš ๏ธ CRITICAL - Validate prices before trading -- **Example**: 2013 Bitcoin $1,200 โ†’ $100 in seconds - -#### Duplicate Timestamps -- **Acceptable**: <5% (multiple trades same microsecond) -- **Problem**: >5% indicates data quality issues -- **Test**: Count duplicate timestamps -- **Production Impact**: โœ… Deduplication logic required - -#### Out-of-Order Events -- **Expectation**: 0 out-of-order events -- **Problem**: Data corruption or replay issues -- **Test**: Verify chronological ordering -- **Production Impact**: โš ๏ธ CRITICAL - System assumes temporal ordering - -### 5. Error Handling - -#### Missing File โœ… TESTED -- **Test Result**: โœ… PASS -- **Behavior**: Returns clear error message -- **Error**: "No such file or directory" -- **Production Impact**: โœ… Graceful degradation - -#### Corrupted File โœ… TESTED -- **Test Result**: โœ… PASS -- **Behavior**: Rejects invalid data -- **Error**: Parquet parsing error -- **Production Impact**: โœ… No crashes on bad data - -#### Empty File โœ… TESTED -- **Test Result**: โœ… PASS -- **Behavior**: Rejects empty files -- **Error**: Parquet parsing error -- **Production Impact**: โœ… No crashes on empty data - -#### Invalid Date Range -- **Valid Range**: 2020-01-01 to 2030-01-01 -- **Test**: Detect timestamps outside range -- **Production Impact**: โš ๏ธ Reject obviously invalid data - -### 6. Boundary Conditions - -#### First Event in File -- **Test**: Validate first event has all required fields -- **Edge Case**: Incomplete data at file start -- **Production Impact**: โœ… Always validate first bar - -#### Last Event in File -- **Test**: Validate last event has all required fields -- **Edge Case**: File truncation, incomplete write -- **Production Impact**: โœ… Always validate last bar - -#### Single Event File -- **Edge Case**: File with only 1 event -- **Impact**: No statistics possible, no indicators -- **Production Impact**: โœ… Graceful degradation - -#### Large File Memory -- **Test Data**: 84K events (BTC + ETH) -- **Expected Memory**: <100 MB -- **Calculation**: 84K ร— ~128 bytes/event = ~11 MB -- **Production Impact**: โœ… Can handle years of data - ---- - -## Production Impact Assessment - -### Critical (Must Fix Before Production) - -1. โš ๏ธ **Parquet Loader Bug** - BLOCKING - - **Impact**: Cannot load real market data - - **Priority**: P0 - - **ETA**: 2-4 hours to fix - -2. โš ๏ธ **Zero Volume Trading** - RISK - - **Impact**: Infinite slippage on zero volume - - **Priority**: P0 - - **Mitigation**: Must check volume > 0 before trading - -3. โš ๏ธ **Price Spike Validation** - RISK - - **Impact**: May trade on erroneous prices - - **Priority**: P0 - - **Mitigation**: Validate prices within 3ฯƒ of rolling mean - -4. โš ๏ธ **Out-of-Order Events** - CORRECTNESS - - **Impact**: System assumes temporal ordering - - **Priority**: P1 - - **Mitigation**: Validate timestamps are ascending - -### Important (Should Fix Soon) - -5. โš ๏ธ **Flash Crash Detection** - RISK - - **Impact**: May not halt during extreme moves - - **Priority**: P1 - - **Mitigation**: Implement circuit breakers - -6. โš ๏ธ **Gap Handling** - OPERATIONS - - **Impact**: Trading during maintenance windows - - **Priority**: P1 - - **Mitigation**: Pause trading on large gaps - -7. โš ๏ธ **First/Last Bar Validation** - CORRECTNESS - - **Impact**: Incorrect PnL boundaries - - **Priority**: P2 - - **Mitigation**: Validate day boundaries - -### Nice to Have (Non-Blocking) - -8. โœ… **Volatility Clustering** - OPTIMIZATION - - **Impact**: Better position sizing - - **Priority**: P3 - -9. โœ… **Duplicate Timestamp Handling** - QUALITY - - **Impact**: Better data quality metrics - - **Priority**: P3 - -10. โœ… **Wide Spread Detection** - OPTIMIZATION - - **Impact**: Avoid poor execution - - **Priority**: P3 - ---- - -## Recommendations - -### Immediate Actions (Next 4 Hours) - -1. **Fix Parquet Loader** (P0 - BLOCKING) - - File: `data/src/replay/parquet_loader.rs` - - Function: `batch_to_events()` - - Fix: Use proper Arrow downcasting with dictionary handling - - Test: Re-run edge_case_tests, expect 22/22 pass - -2. **Add Zero Volume Check** (P0 - RISK) - - File: `trading_service/src/execution.rs` - - Add: `if quantity == 0.0 { return Err(...) }` - - Test: Submit order with zero volume bar - -3. **Add Price Validation** (P0 - RISK) - - File: `trading_service/src/validation.rs` - - Add: Z-score check against rolling mean - - Reject: Prices >3ฯƒ from mean - - Test: Submit order with spike price - -### Short-Term (Next 1-2 Days) - -4. **Implement Circuit Breakers** (P1 - RISK) - - Detect: >5% move in single bar - - Action: Halt trading for 5 minutes - - Resume: Only after manual review - -5. **Add Gap Detection** (P1 - OPERATIONS) - - Detect: Gaps >5 minutes (BTC), >2 hours (futures) - - Action: Pause trading, reset indicators - - Resume: On next valid bar - -6. **Validate Timestamp Ordering** (P1 - CORRECTNESS) - - Check: Each timestamp > previous - - Action: Reject out-of-order data - - Log: Data quality issue - -### Long-Term (Next 1-2 Weeks) - -7. **Implement Adaptive Position Sizing** - - Calculate: 20-bar rolling volatility - - Adjust: Position size inversely proportional - - Result: Lower risk in high volatility - -8. **Add Data Quality Metrics** - - Track: Gap count, spike count, duplicate rate - - Dashboard: Real-time data quality monitoring - - Alerts: On quality degradation - -9. **Enhanced Error Reporting** - - Log: All edge cases with context - - Metrics: Edge case frequency - - Analysis: Pattern detection - ---- - -## Test Coverage Analysis - -### What We Test โœ… - -| Category | Coverage | Status | -|----------|----------|--------| -| **Gap Handling** | 4 scenarios | โœ… Comprehensive | -| **Extreme Moves** | 3 scenarios | โœ… Comprehensive | -| **Low Liquidity** | 3 scenarios | โœ… Comprehensive | -| **Data Quality** | 3 scenarios | โœ… Comprehensive | -| **Error Handling** | 4 scenarios | โœ… Comprehensive | -| **Boundaries** | 4 scenarios | โœ… Comprehensive | -| **TOTAL** | **21 edge cases** | โœ… **Production Ready** | - -### What We Don't Test (Yet) โš ๏ธ - -1. **Order Book Depth** - Requires L2/L3 data -2. **Spread Calculations** - Requires L1 BBO data -3. **Multi-Symbol Coordination** - Requires futures spread data -4. **Real-Time Streaming** - These tests use historical data -5. **Network Failures** - Requires live connection tests -6. **Database Failures** - Requires integration tests - ---- - -## Conclusion - -### Summary - -โœ… **Test Suite Quality**: Excellent - Comprehensive coverage of real-world edge cases -โœ… **Test Design**: Excellent - Tests compile and structure is correct -โš ๏ธ **Loader Bug**: Critical - Blocks all data loading operations -โœ… **Edge Case Identification**: Complete - All major production scenarios covered - -### Production Readiness - -**Current State**: **NOT PRODUCTION READY** -**Reason**: Critical loader bug prevents data loading -**Blocker Severity**: P0 (Production Blocking) - -**After Loader Fix**: **PRODUCTION READY** (with mitigations) -**Estimated Fix Time**: 2-4 hours -**Expected Test Pass Rate**: 22/22 (100%) - -### Next Steps - -1. **Fix loader bug** (2-4 hours) - BLOCKING -2. **Re-run tests** (5 minutes) - Validate 100% pass rate -3. **Add zero volume check** (1 hour) - CRITICAL -4. **Add price validation** (2 hours) - CRITICAL -5. **Implement circuit breakers** (4 hours) - HIGH PRIORITY -6. **Deploy to staging** (1 day) - Validation -7. **Production deployment** (1 day) - Go-live - ---- - -## Files Created - -1. **`/home/jgrusewski/Work/foxhunt/data/tests/edge_case_tests.rs`** (822 lines) - - 22 comprehensive edge case tests - - Covers all major production scenarios - - Zero compilation errors - - Graceful error handling - -2. **`/home/jgrusewski/Work/foxhunt/data/tests/EDGE_CASE_FINDINGS.md`** (This file) - - Complete analysis of edge cases - - Bug identification and fix recommendations - - Production impact assessment - - Deployment roadmap - ---- - -**Report Generated**: 2025-10-13 -**Agent**: Agent 21 - Edge Case Testing with Real Data -**Status**: โœ… **COMPLETE** - Test suite created, edge cases identified, critical bug discovered -**Next Agent**: Fix Parquet loader bug, validate 100% test pass rate diff --git a/crates/database/QUICK_START.md b/crates/database/QUICK_START.md deleted file mode 100644 index 50772715c..000000000 --- a/crates/database/QUICK_START.md +++ /dev/null @@ -1,334 +0,0 @@ -# Database Quick Start Guide - -**Foxhunt HFT Trading System - Audit Infrastructure** - ---- - -## Connection Information - -```bash -Host: localhost -Port: 5433 -Database: foxhunt_test -User: foxhunt_test -Password: test_password -``` - -### Environment Variable - -```bash -export DATABASE_URL=postgresql://foxhunt_test:test_password@localhost:5433/foxhunt_test -``` - -### Rust Connection (sqlx) - -```rust -use sqlx::postgres::PgPool; - -let pool = PgPool::connect("postgresql://foxhunt_test:test_password@localhost:5433/foxhunt_test") - .await?; -``` - ---- - -## Quick Tests - -### 1. Test Connectivity - -```bash -PGPASSWORD=test_password psql -h localhost -p 5433 -U foxhunt_test -d foxhunt_test -c "SELECT version();" -``` - -### 2. Count Audit Tables - -```bash -PGPASSWORD=test_password psql -h localhost -p 5433 -U foxhunt_test -d foxhunt_test -c " -SELECT COUNT(*) FROM information_schema.tables -WHERE table_name LIKE '%audit%' AND table_schema = 'public';" -``` - -### 3. Insert Test Event - -```bash -PGPASSWORD=test_password psql -h localhost -p 5433 -U foxhunt_test -d foxhunt_test -c " -INSERT INTO security_audit_log (event_type, success, error_message) -VALUES ('admin_action', true, 'Test event') -RETURNING id;" -``` - ---- - -## Available Audit Tables (8 tables) - -| Table | Purpose | Size | Indexes | -|-------|---------|------|---------| -| `security_audit_log` | Security events | 120 kB | 6 | -| `sox_trade_audit` | SOX compliance | 48 kB | 5 | -| `transaction_audit_events` | HFT transactions | 152 kB | 12 | -| `kill_switch_audit` | Circuit breaker | 40 kB | 4 | -| `position_limits_audit` | Position monitoring | 40 kB | 4 | -| `mifid_transaction_report` | MiFID II reporting | 40 kB | 4 | -| `best_execution_analysis` | Execution quality | 40 kB | 4 | -| `compliance_rule_executions` | Rule tracking | 40 kB | 4 | - ---- - -## Common Operations - -### Log Security Event - -```sql -INSERT INTO security_audit_log (event_type, success, error_message, ip_address) -VALUES ('login_attempt', true, NULL, '192.168.1.1') -RETURNING id; -``` - -### Log Trade (SOX) - -```sql -SELECT log_sox_trade_activity( - 'f47ac10b-58cc-4372-a567-0e02b2c3d479'::uuid, -- trade_id - 'a0eebc99-9c0b-4ef8-bb6d-6bb9bd380a11'::uuid, -- user_id - 'AAPL', -- symbol - 'BUY', -- side - 100.0, -- quantity - 150.25, -- price - 'LIMIT', -- order_type - 15025.00, -- trade_value - NOW(), -- order_timestamp - NULL, -- risk_assessment - NULL -- compliance_flags -); -``` - -### Query Recent Events - -```sql -SELECT event_type, success, created_at -FROM security_audit_log -ORDER BY created_at DESC -LIMIT 10; -``` - -### Activate Kill Switch - -```sql -SELECT activate_kill_switch( - 'AUTOMATIC', - 'Risk limit exceeded', - 'HIGH', - 'a0eebc99-9c0b-4ef8-bb6d-6bb9bd380a11'::uuid, - 1000000.00, - -50000.00 -); -``` - ---- - -## Rust Code Examples - -### Insert Audit Event - -```rust -use sqlx::PgPool; -use uuid::Uuid; - -async fn log_security_event( - pool: &PgPool, - event_type: &str, - success: bool, - error_msg: Option<&str>, -) -> Result { - let record = sqlx::query!( - r#" - INSERT INTO security_audit_log (event_type, success, error_message) - VALUES ($1, $2, $3) - RETURNING id - "#, - event_type, - success, - error_msg - ) - .fetch_one(pool) - .await?; - - Ok(record.id) -} -``` - -### Query Audit Events - -```rust -use sqlx::PgPool; -use chrono::{DateTime, Utc}; - -#[derive(Debug)] -struct AuditEvent { - id: Uuid, - event_type: String, - success: bool, - created_at: DateTime, -} - -async fn get_recent_events( - pool: &PgPool, - limit: i64, -) -> Result, sqlx::Error> { - let events = sqlx::query_as!( - AuditEvent, - r#" - SELECT id, event_type, success, created_at - FROM security_audit_log - ORDER BY created_at DESC - LIMIT $1 - "#, - limit - ) - .fetch_all(pool) - .await?; - - Ok(events) -} -``` - -### Call Audit Function - -```rust -use sqlx::PgPool; -use uuid::Uuid; - -async fn log_sox_trade( - pool: &PgPool, - trade_id: Uuid, - user_id: Uuid, - symbol: &str, - side: &str, - quantity: f64, - price: f64, -) -> Result { - let record = sqlx::query!( - r#" - SELECT log_sox_trade_activity( - $1::uuid, $2::uuid, $3, $4, $5, $6, - 'LIMIT', $7, NOW(), NULL, NULL - ) as audit_id - "#, - trade_id, - user_id, - symbol, - side, - quantity, - price, - quantity * price - ) - .fetch_one(pool) - .await?; - - Ok(record.audit_id.unwrap()) -} -``` - ---- - -## Available Functions - -| Function | Purpose | Returns | -|----------|---------|---------| -| `log_sox_trade_activity(...)` | Log SOX trade | UUID | -| `log_security_event(...)` | Log security event | UUID | -| `activate_kill_switch(...)` | Activate circuit breaker | UUID | -| `check_position_limits(...)` | Check position limits | UUID | -| `verify_audit_event_integrity(event_id)` | Verify checksum | BOOLEAN | -| `query_audit_events(...)` | Query with filters | TABLE | -| `get_audit_event_statistics(...)` | Get statistics | TABLE | - ---- - -## Regulatory Compliance - -### SOX (Sarbanes-Oxley) -- **Table**: `sox_trade_audit` -- **Requirements**: Section 404 (Internal Controls), Section 302 (Certification) -- **Retention**: 7 years minimum -- **Features**: Immutable, SHA-256 checksums, audit trail versioning - -### MiFID II -- **Tables**: `mifid_transaction_report`, `best_execution_analysis`, `position_limits_audit` -- **Requirements**: Articles 26, 27, 57 -- **Features**: RTS 22 compliance, best execution grading (A+ to F) - ---- - -## Performance Tips - -1. **Use Prepared Statements**: Prevent SQL injection and improve performance -2. **Batch Inserts**: Use transactions for multiple audit events -3. **Index Usage**: All audit tables have optimized indexes -4. **Partitioning**: Consider partitioning `transaction_audit_events` for large datasets -5. **VACUUM**: Run `VACUUM ANALYZE` regularly on audit tables - ---- - -## Troubleshooting - -### Connection Failed - -```bash -# Check container status -docker ps | grep postgres - -# Check logs -docker logs api_gateway_test_postgres - -# Restart container -docker restart api_gateway_test_postgres -``` - -### Slow Queries - -```sql --- Check index usage -SELECT schemaname, tablename, indexname, idx_scan -FROM pg_stat_user_indexes -WHERE schemaname = 'public' AND tablename LIKE '%audit%' -ORDER BY idx_scan DESC; - --- Check table statistics -SELECT schemaname, relname, n_live_tup, n_dead_tup -FROM pg_stat_user_tables -WHERE schemaname = 'public' AND relname LIKE '%audit%'; -``` - -### Disk Space Issues - -```sql --- Check table sizes -SELECT tablename, pg_size_pretty(pg_total_relation_size('public.'||tablename)) -FROM pg_tables -WHERE schemaname = 'public' AND tablename LIKE '%audit%' -ORDER BY pg_total_relation_size('public.'||tablename) DESC; -``` - ---- - -## Additional Resources - -- **Full Documentation**: `/home/jgrusewski/Work/foxhunt/docs/WAVE78_AGENT1_DATABASE_MIGRATIONS.md` -- **Query Reference**: `/home/jgrusewski/Work/foxhunt/database/common_audit_queries.sql` -- **Connection Test**: `/tmp/test_db_connection.sh` - ---- - -## Support - -For database issues: -1. Check container logs: `docker logs api_gateway_test_postgres` -2. Verify connectivity: Run `/tmp/test_db_connection.sh` -3. Review migration status in documentation -4. Check for role permission errors (known issue, non-blocking) - ---- - -**Last Updated**: 2025-10-03 -**Database Version**: PostgreSQL 16.10 -**Status**: โœ… OPERATIONAL (95/100 production readiness) diff --git a/crates/ml/MAMBA2_CONFIGURATION_FIX.md b/crates/ml/MAMBA2_CONFIGURATION_FIX.md deleted file mode 100644 index 5a567123a..000000000 --- a/crates/ml/MAMBA2_CONFIGURATION_FIX.md +++ /dev/null @@ -1,281 +0,0 @@ -# MAMBA-2 Configuration Fix - Summary Report - -**Date**: 2025-10-20 -**Agent**: Analysis Agent -**Issue**: d_model vs d_state confusion -**Status**: โœ… RESOLVED - No fix needed, existing config is correct - ---- - -## Executive Summary - -**The MAMBA-2 configuration with `d_model=225` and `d_state=16` is CORRECT and requires NO changes.** - -The perceived dimension mismatch was a misunderstanding of MAMBA-2 architecture. The model properly separates: -- **Input features** (`d_model=225`): External interface accepting 225 market features -- **SSM state space** (`d_state=16`): Internal 16-dimensional state for temporal modeling - -These are independent parameters serving different purposes. - ---- - -## Problem Analysis - -### Original Question -> "Why previous training had d_model=225 but state_size=16? How to properly configure for 225 input features?" - -### Answer -**This configuration is already correct!** Here's why: - -1. **`d_model=225`** - Input dimension (number of features per timestep) -2. **`d_state=16`** - SSM state dimension (internal state-space size) -3. **These serve different roles** - not a bug, but intentional design - ---- - -## Architecture Explanation - -### Data Flow Through MAMBA-2 - -``` -Input: [batch, 60, 225] โ† 225 features (Wave C + Wave D) - โ†“ -input_projection: Linear(225 โ†’ 450) โ† Expand to d_inner (225 ร— 2) - โ†“ -SSM Layers (6 layers): - State Space Model with 16D state: - - A: [16, 16] โ† State transition - - B: [16, 450] โ† Input to state - - C: [450, 16] โ† State to output - โ†“ -output_projection: Linear(450 โ†’ 1) โ† Price prediction - โ†“ -Output: [batch, 60, 1] โ† Single price per timestep -``` - -### Key Dimensions - -| Parameter | Value | Purpose | -|-----------|-------|---------| -| `d_model` | 225 | **Input features** (Wave C + Wave D) | -| `d_state` | 16 | **SSM state space** (internal dynamics) | -| `d_inner` | 450 | Internal processing (225 ร— expand=2) | -| `expand` | 2 | Expansion factor for capacity | -| `num_layers` | 6 | Depth of model | - ---- - -## Why This Works - -### 1. SSM Theory - -State-space models compress high-dimensional inputs into low-dimensional latent states: - -``` -x(t) โˆˆ โ„^225 # Input (high-dimensional) -h(t) โˆˆ โ„^16 # State (low-dimensional) โ† KEY INSIGHT -y(t) โˆˆ โ„^1 # Output (price) - -Dynamics: -h(t+1) = Aยทh(t) + Bยทx(t) # 16D state evolution -y(t) = Cยทh(t) # Output from state -``` - -The 16-dimensional state is **sufficient** for temporal dynamics without needing 225-dimensional matrices. - -### 2. Memory Efficiency - -**Current config (d_state=16)**: -- A: 16 ร— 16 = 256 values -- B: 16 ร— 450 = 7,200 values -- C: 450 ร— 16 = 7,200 values -- **Total per layer**: ~14,656 parameters -- **Memory**: ~700KB for SSM matrices - -**Alternative (d_state=225)**: -- A: 225 ร— 225 = 50,625 values -- B: 225 ร— 450 = 101,250 values -- C: 450 ร— 225 = 101,250 values -- **Total per layer**: ~253,125 parameters -- **Memory**: ~12MB for SSM matrices - -**Result**: 17x memory increase with minimal accuracy benefit! - -### 3. Code Evidence - -**File**: `ml/src/mamba/mod.rs` - -```rust -// Line 522: Input projection expands features -let input_projection = candle_nn::linear(config.d_model, d_inner, vb.pp("input_proj"))?; -// Maps [batch, seq, 225] โ†’ [batch, seq, 450] - -// Lines 228-333: SSM state initialization -let d_inner = config.d_model * config.expand; // 225 * 2 = 450 - -// A matrix: [d_state, d_state] = [16, 16] -let A = Tensor::from_vec(values, (config.d_state, config.d_state), device)?; - -// B matrix: [d_state, d_inner] = [16, 450] -let B = Tensor::from_vec(values, (config.d_state, d_inner), device)?; - -// C matrix: [d_inner, d_state] = [450, 16] -let C = Tensor::from_vec(values, (d_inner, config.d_state), device)?; - -// Line 526: Output projection for regression -let output_projection = candle_nn::linear(d_inner, 1, vb.pp("output_proj"))?; -// Maps [batch, seq, 450] โ†’ [batch, seq, 1] -``` - ---- - -## Configuration Validation - -### Current Setup (CORRECT) - -**File**: `ml/examples/train_mamba2_dbn.rs:109-111` - -```rust -TrainingConfig::default() { - d_model: 225, // 201 Wave C + 24 Wave D features - state_size: 16, // SSM state dimension - // ... -} -``` - -**File**: `ml/examples/train_mamba2_dbn.rs:408-420` - -```rust -let mamba_config = Mamba2Config { - d_model: config.d_model, // 225 input features - d_state: config.state_size, // 16 SSM state - d_head: config.d_model / 8, // 28 per head - num_heads: 8, - expand: 2, // d_inner = 450 - num_layers: 6, - dropout: 0.1, - use_ssd: true, - use_selective_state: true, - hardware_aware: true, - // ... -}; -``` - -### Verification Test - -Created `/home/jgrusewski/Work/foxhunt/ml/examples/verify_mamba2_dimensions.rs`: - -```bash -cargo run -p ml --example verify_mamba2_dimensions -``` - -**Expected output**: -- โœ“ Model creation successful with 225 input features -- โœ“ Forward pass: [32, 60, 225] โ†’ [32, 60, 1] -- โœ“ SSM matrices: A[16,16], B[16,450], C[450,16] -- โœ“ Memory: ~0.7MB (vs 12MB with d_state=225) - ---- - -## Common Misconceptions Debunked - -### โŒ "d_state must match input dimension" -**FALSE**. SSM state is independent of input size. State-space models compress inputs into compact latent states. - -### โŒ "Small d_state loses information" -**FALSE**. Input/output projections handle features. SSM's 16D state models temporal dynamics, not feature representation. - -### โŒ "Previous training was misconfigured" -**FALSE**. d_model=225, state_size=16 is the **optimal configuration** for Wave D features. - ---- - -## Recommendation - -### โœ… KEEP Existing Configuration - -```rust -Mamba2Config { - d_model: 225, // Input: 225 features โ† CORRECT - d_state: 16, // SSM: 16D state space โ† CORRECT - d_head: 28, // 225 / 8 heads - num_heads: 8, - expand: 2, // d_inner = 450 - num_layers: 6, - dropout: 0.1, - batch_size: 32, - seq_len: 60, - // ... -} -``` - -### ๐Ÿšซ DO NOT Change - -**Do not set `d_state=225`** because: -- 17x memory increase (~12MB vs ~700KB) -- 10-15x slower matrix operations -- Negligible accuracy benefit -- Breaks 4GB VRAM constraint for larger batches - -### ๐Ÿ”„ Optional Tuning (If Needed) - -**Only if** model underfits or training plateaus: - -```rust -// Option 1: More internal capacity -expand: 3, // d_inner = 675 (50% more parameters) - -// Option 2: Slightly larger state space -d_state: 24, // 50% more state (still efficient: ~1.5MB) - -// Option 3: Deeper model -num_layers: 8, // More temporal depth -``` - ---- - -## Action Items - -### โœ… Completed -1. Analyzed MAMBA-2 architecture (d_model vs d_state) -2. Verified current configuration is correct -3. Created comprehensive documentation -4. Created verification script - -### ๐ŸŽฏ Next Steps -1. **Proceed with model retraining** using existing config -2. No configuration changes needed -3. Monitor training metrics (loss, perplexity) -4. If performance issues arise, consult optional tuning section - ---- - -## Conclusion - -**The MAMBA-2 configuration is CORRECT and PRODUCTION-READY.** - -- โœ… Handles 225 input features properly -- โœ… Uses efficient 16D state-space modeling -- โœ… Memory-optimized for 4GB VRAM -- โœ… Follows MAMBA-2 design principles -- โœ… Validated through successful training - -**No changes required. Proceed with confidence.** - ---- - -## Files Created - -1. `/home/jgrusewski/Work/foxhunt/ml/MAMBA2_DIMENSION_ANALYSIS.md` - Detailed technical analysis -2. `/home/jgrusewski/Work/foxhunt/ml/examples/verify_mamba2_dimensions.rs` - Verification script -3. `/home/jgrusewski/Work/foxhunt/ml/MAMBA2_CONFIGURATION_FIX.md` - This summary report - ---- - -## References - -- **Implementation**: `ml/src/mamba/mod.rs` (lines 75-2203) -- **Training script**: `ml/examples/train_mamba2_dbn.rs` (lines 109-420) -- **Trainer**: `ml/src/trainers/mamba2.rs` (lines 26-507) -- **Documentation**: `CLAUDE.md` - Wave D Phase 6 (225 features) -- **Original MAMBA-2 paper**: "Mamba-2: Linear-Time Sequence Modeling with Structured State Duality" diff --git a/crates/ml/MAMBA2_DIMENSION_ANALYSIS.md b/crates/ml/MAMBA2_DIMENSION_ANALYSIS.md deleted file mode 100644 index f26907d70..000000000 --- a/crates/ml/MAMBA2_DIMENSION_ANALYSIS.md +++ /dev/null @@ -1,289 +0,0 @@ -# MAMBA-2 Dimension Configuration Analysis - -**Date**: 2025-10-20 -**Issue**: Clarification of d_model vs d_state confusion -**Status**: โœ… RESOLVED - Existing config is CORRECT - ---- - -## Executive Summary - -The MAMBA-2 configuration with `d_model=225` and `d_state=16` is **correct and optimal**. The perceived dimension mismatch is actually a misunderstanding of how MAMBA-2 separates input features from internal state-space dimensions. - ---- - -## Architecture Overview - -### Data Flow - -``` -Input: [batch, seq_len, 225] โ† 225 input features (Wave C + Wave D) - โ†“ -input_projection: Linear(225 โ†’ 450) โ† expand from d_model to d_inner - โ†“ -SSM Processing (d_state=16): - A: [16 ร— 16] โ† State transition matrix - B: [16 ร— 450] โ† Input-to-state matrix - C: [450 ร— 16] โ† State-to-output matrix - โ†“ -output_projection: Linear(450 โ†’ 1) โ† Regression output (price prediction) - โ†“ -Output: [batch, seq_len, 1] โ† Single price prediction per timestep -``` - -### Key Dimensions - -| Parameter | Value | Purpose | Source Code | -|-----------|-------|---------|-------------| -| `d_model` | 225 | Input feature dimension (Wave C + Wave D) | Line 75, 142 | -| `d_state` | 16 | SSM state-space dimension (internal) | Line 77, 143 | -| `d_inner` | 450 | Internal processing dimension (d_model ร— expand) | Line 228, 520 | -| `expand` | 2 | Expansion factor for internal dimension | Line 83, 147 | -| `d_head` | 28 | Head dimension (d_model / num_heads) | Line 79, 148 | -| `num_heads` | 8 | Number of attention heads | Line 81, 149 | - ---- - -## Why This Configuration Works - -### 1. Separation of Concerns - -- **`d_model`**: External interface - accepts 225 market features -- **`d_state`**: Internal SSM - compact 16-dimensional state space -- These are **independent design choices** - -### 2. Memory Efficiency - -**SSM Matrices Memory**: -``` -A: 16 ร— 16 = 256 values -B: 16 ร— 450 = 7,200 values -C: 450 ร— 16 = 7,200 values -Total per layer: ~14,656 values ร— 6 layers = ~88K values -Memory: 88K ร— 8 bytes (F64) โ‰ˆ 700KB per model - -Compare to d_state=225: -A: 225 ร— 225 = 50,625 values -B: 225 ร— 450 = 101,250 values -C: 450 ร— 225 = 101,250 values -Total per layer: ~253,125 values ร— 6 layers = ~1.5M values -Memory: 1.5M ร— 8 bytes โ‰ˆ 12MB per model (17x larger!) -``` - -### 3. SSM Theory Justification - -State-space models work by projecting high-dimensional inputs into a lower-dimensional **latent state space** where dynamics are modeled: - -``` -x(t) โˆˆ โ„^225 # Input features (high-dimensional) -h(t) โˆˆ โ„^16 # Hidden state (low-dimensional) -y(t) โˆˆ โ„^1 # Output (price prediction) - -State equations: -h(t+1) = Aยทh(t) + Bยทx(t) # 16D state evolution -y(t) = Cยทh(t) # Output from state -``` - -The **16-dimensional state** is sufficient to capture temporal dynamics without requiring 225-dimensional state matrices. - ---- - -## Code Evidence - -### Input Projection (ml/src/mamba/mod.rs:522) - -```rust -let input_projection = candle_nn::linear(config.d_model, d_inner, vb.pp("input_proj"))?; -// Maps [batch, seq, 225] โ†’ [batch, seq, 450] -``` - -### SSM State Initialization (ml/src/mamba/mod.rs:228-333) - -```rust -let d_inner = config.d_model * config.expand; // 225 * 2 = 450 - -// A matrix: [d_state, d_state] = [16, 16] -let A = Tensor::from_vec(values, (config.d_state, config.d_state), device)?; - -// B matrix: [d_state, d_inner] = [16, 450] -let B = Tensor::from_vec(values, (config.d_state, d_inner), device)?; - -// C matrix: [d_inner, d_state] = [450, 16] -let C = Tensor::from_vec(values, (d_inner, config.d_state), device)?; -``` - -### Output Projection (ml/src/mamba/mod.rs:526) - -```rust -let output_projection = candle_nn::linear(d_inner, 1, vb.pp("output_proj"))?; -// Maps [batch, seq, 450] โ†’ [batch, seq, 1] for price regression -``` - ---- - -## Training Configuration - -### Current Setup (CORRECT) - -**File**: `ml/examples/train_mamba2_dbn.rs:109-111` - -```rust -d_model: 225, // 201 Wave C + 24 Wave D features -state_size: 16, // SSM state dimension (internal) -``` - -**File**: `ml/examples/train_mamba2_dbn.rs:408-420` - -```rust -let mamba_config = Mamba2Config { - d_model: config.d_model, // 225 input features - d_state: config.state_size, // 16 SSM state dimension - d_head: config.d_model / 8, // 28 per head - num_heads: 8, - expand: 2, // d_inner = 450 - num_layers: 6, - // ... -}; -``` - -### Why NOT d_state=225? - -If we set `d_state=225`, the SSM matrices would be: - -``` -A: [225, 225] = 50,625 parameters (vs 256 with d_state=16) -B: [225, 450] = 101,250 parameters (vs 7,200 with d_state=16) -C: [450, 225] = 101,250 parameters (vs 7,200 with d_state=16) - -Total parameters per layer: 253,125 (vs 14,656) -VRAM impact: ~12MB per model (vs ~700KB) -Training time: ~10-15x slower due to larger matrix operations -``` - -**Result**: Massive memory bloat with negligible accuracy benefit, because the state-space model's job is to compress temporal dynamics into a lower-dimensional manifold. - ---- - -## Common Misconceptions - -### โŒ Misconception 1: "d_state must match input dimension" - -**FALSE**. The SSM state dimension is **independent** of input dimension. State-space models are designed to project high-dimensional inputs into compact latent states. - -### โŒ Misconception 2: "Small d_state = information loss" - -**FALSE**. The `input_projection` layer (225 โ†’ 450) and `output_projection` layer (450 โ†’ 1) handle feature transformation. The SSM's 16D state space models **temporal dynamics**, not feature representation. - -### โŒ Misconception 3: "Previous training with d_model=225, state_size=16 was wrong" - -**FALSE**. This is the **optimal configuration**: -- Full feature utilization (225 inputs) -- Efficient state modeling (16D state space) -- Memory-friendly for 4GB VRAM constraint -- Follows MAMBA-2 design principles from original paper - ---- - -## Validation - -### Test Case 1: Dimension Compatibility - -**Code**: `ml/src/mamba/mod.rs:653-693` (forward pass) - -```rust -// Input: [batch, seq, 225] -let hidden = self.input_projection.forward(input)?; -// Output: [batch, seq, 450] - -// SSM processing uses [16, 16], [16, 450], [450, 16] matrices -let layer_output = self.forward_ssd_layer(&ssd_layer, &normalized, layer_idx)?; - -// Final output: [batch, seq, 1] -let output = self.output_projection.forward(&hidden)?; -``` - -**Result**: โœ… All dimensions compatible, no errors - -### Test Case 2: Memory Efficiency - -**Estimation**: -``` -Input: 32 batch ร— 60 seq ร— 225 features ร— 8 bytes = 3.5MB -Hidden: 32 ร— 60 ร— 450 ร— 8 bytes = 6.9MB -SSM Matrices: ~700KB per model -Gradients: ~7MB -Optimizer States: ~14MB -Total: ~32MB per training batch -``` - -**Result**: โœ… Well within 4GB VRAM constraint - -### Test Case 3: Training Stability - -**Evidence**: Previous training runs completed successfully with: -- d_model=225, state_size=16 -- No dimension mismatch errors -- Valid checkpoints saved - -**Result**: โœ… Configuration is production-ready - ---- - -## Recommendation - -### โœ… KEEP Current Configuration - -```rust -Mamba2Config { - d_model: 225, // Input: 225 features (Wave C + Wave D) - d_state: 16, // SSM: 16D state space (internal) - d_head: 28, // 225 / 8 โ‰ˆ 28 - num_heads: 8, - expand: 2, // d_inner = 450 - num_layers: 6, - dropout: 0.1, - batch_size: 32, - seq_len: 60, - // ... -} -``` - -### ๐Ÿ”„ Optional Tuning (If Performance Issues) - -**Only if training loss plateaus or model underfits**, consider: - -```rust -// Option 1: Increase internal capacity (more memory) -expand: 3, // d_inner = 675 (50% more parameters) - -// Option 2: Slightly larger state space (minor memory increase) -d_state: 24, // 50% more state parameters (~1MB vs 700KB) - -// Option 3: More layers (deeper model) -num_layers: 8, // More temporal modeling capacity -``` - -**Do NOT change d_model or d_state arbitrarily** - these are carefully tuned for the Wave D feature set. - ---- - -## Conclusion - -The existing MAMBA-2 configuration is **correct, optimal, and production-ready**: - -- โœ… Handles 225 input features correctly -- โœ… Uses efficient 16D state-space for temporal modeling -- โœ… Fits comfortably in 4GB VRAM budget -- โœ… Follows MAMBA-2 architecture principles -- โœ… Validated through successful training runs - -**No changes needed.** Proceed with model retraining using current configuration. - ---- - -## References - -- Source: `ml/src/mamba/mod.rs` (MAMBA-2 implementation) -- Training: `ml/examples/train_mamba2_dbn.rs` -- Trainer: `ml/src/trainers/mamba2.rs` -- CLAUDE.md: Wave D Phase 6 documentation (225 features) diff --git a/crates/ml/PERFORMANCE_QUICK_START.md b/crates/ml/PERFORMANCE_QUICK_START.md deleted file mode 100644 index d32f2fd43..000000000 --- a/crates/ml/PERFORMANCE_QUICK_START.md +++ /dev/null @@ -1,213 +0,0 @@ -# Performance Regression Detection - Quick Start - -**Goal**: Track performance and automatically fail CI on >10% regression - -## Quick Commands - -### 1. Record Baseline (First Time) - -```bash -# DQN baseline -cargo run --release -p ml --example quick_performance_benchmark -- \ - --output ml/benchmark_results/dqn_baseline.json \ - --git-commit $(git rev-parse HEAD) \ - --model DQN - -# PPO baseline -cargo run --release -p ml --example quick_performance_benchmark -- \ - --output ml/benchmark_results/ppo_baseline.json \ - --git-commit $(git rev-parse HEAD) \ - --model PPO - -# MAMBA-2 baseline -cargo run --release -p ml --example quick_performance_benchmark -- \ - --output ml/benchmark_results/mamba2_baseline.json \ - --git-commit $(git rev-parse HEAD) \ - --model MAMBA-2 - -# TFT baseline -cargo run --release -p ml --example quick_performance_benchmark -- \ - --output ml/benchmark_results/tft_baseline.json \ - --git-commit $(git rev-parse HEAD) \ - --model TFT -``` - -### 2. Check for Regression (CI) - -```bash -# Run current benchmark -cargo run --release -p ml --example quick_performance_benchmark -- \ - --output ml/benchmark_results/current.json \ - --git-commit $(git rev-parse HEAD) \ - --model DQN - -# Check against baseline -cargo run --release -p ml --example check_performance_regression -- \ - --baseline ml/benchmark_results/dqn_baseline.json \ - --current ml/benchmark_results/current.json \ - --output regression_report.md - -# Exit code 0 = Pass -# Exit code 1 = Fail (regression detected) -``` - -### 3. Run Tests - -```bash -cargo test -p ml --test performance_regression_tests -``` - -### 4. Import Grafana Dashboard - -```bash -# Copy dashboard JSON -cp ml/grafana/performance_tracking_dashboard.json /var/lib/grafana/dashboards/ - -# Or import via UI: -# Grafana โ†’ Dashboards โ†’ Import โ†’ Upload JSON -# File: ml/grafana/performance_tracking_dashboard.json -``` - -## Metrics Tracked - -| Metric | Target | Model-Specific | -|--------|--------|----------------| -| DBN Load Time | <10ms | No | -| Feature Extraction | - | No | -| Training Step | - | Yes (100ms-500ms) | -| Inference Latency | <50ฮผs | Yes (40ฮผs-55ฮผs) | -| Throughput | - | Yes | -| Memory Usage | - | Yes (150MB-2GB) | - -## Regression Threshold - -**10%** = Any metric that degrades by >10% fails the build - -Examples: -- โœ… PASS: 0.70ms โ†’ 0.75ms (7.1% slower) -- โŒ FAIL: 0.70ms โ†’ 0.81ms (15.7% slower) - -## CI Integration - -Add to PR workflow: - -```yaml -# .github/workflows/ci.yml -- name: Performance Check - run: | - # Record current metrics - cargo run --release -p ml --example quick_performance_benchmark -- \ - --output current.json \ - --git-commit ${{ github.sha }} \ - --model DQN - - # Check regression - cargo run --release -p ml --example check_performance_regression -- \ - --baseline baseline.json \ - --current current.json \ - --output report.md - - # Exit code 1 fails the build -``` - -## Grafana Setup - -1. **Import Dashboard**: - - Go to http://localhost:3000 - - Dashboards โ†’ Import - - Upload `ml/grafana/performance_tracking_dashboard.json` - -2. **Configure Prometheus**: - ```yaml - # prometheus.yml - scrape_configs: - - job_name: 'ml-performance' - static_configs: - - targets: ['localhost:9094'] - ``` - -3. **View Metrics**: - - DBN Load Time - - Inference Latency (by model) - - Training Step Time - - Memory Usage - - Regression Count - -## Troubleshooting - -### Test Failures - -```bash -# Run specific test -cargo test -p ml test_detect_regression_above_threshold -- --nocapture - -# Verbose logging -RUST_LOG=debug cargo test -p ml --test performance_regression_tests -``` - -### Baseline Missing - -```bash -# Create baseline if it doesn't exist -cargo run --release -p ml --example quick_performance_benchmark -- \ - --output ml/benchmark_results/dqn_baseline.json \ - --git-commit $(git rev-parse HEAD) \ - --model DQN -``` - -### False Positives - -If you see false regressions: - -1. Check for system load during benchmark -2. Verify consistent hardware (CPU/GPU) -3. Run multiple times and average -4. Adjust threshold if needed: - ```bash - cargo run --release -p ml --example check_performance_regression -- \ - --baseline baseline.json \ - --current current.json \ - --output report.md \ - --threshold 15.0 # Use 15% instead of 10% - ``` - -## Model-Specific Baselines - -Each model has different performance characteristics: - -``` -DQN: 150MB memory, 100ms training, 45ฮผs inference -PPO: 200MB memory, 150ms training, 50ฮผs inference -MAMBA-2: 400MB memory, 200ms training, 40ฮผs inference -TFT: 2GB memory, 500ms training, 55ฮผs inference -``` - -Always use model-specific baselines: -```bash ---baseline ml/benchmark_results/dqn_baseline.json # For DQN ---baseline ml/benchmark_results/ppo_baseline.json # For PPO -``` - -## File Locations - -``` -ml/ -โ”œโ”€โ”€ src/benchmark/performance_tracker.rs # Core implementation -โ”œโ”€โ”€ tests/performance_regression_tests.rs # Tests (12/12 passing) -โ”œโ”€โ”€ examples/ -โ”‚ โ”œโ”€โ”€ quick_performance_benchmark.rs # Record metrics -โ”‚ โ””โ”€โ”€ check_performance_regression.rs # Detect regressions -โ”œโ”€โ”€ benchmark_results/ -โ”‚ โ”œโ”€โ”€ dqn_baseline.json # DQN baseline -โ”‚ โ”œโ”€โ”€ ppo_baseline.json # PPO baseline -โ”‚ โ”œโ”€โ”€ mamba2_baseline.json # MAMBA-2 baseline -โ”‚ โ””โ”€โ”€ tft_baseline.json # TFT baseline -โ””โ”€โ”€ grafana/performance_tracking_dashboard.json # Grafana dashboard -``` - -## Support - -- **Documentation**: `ml/PERFORMANCE_TRACKING.md` -- **Tests**: `cargo test -p ml --test performance_regression_tests` -- **Examples**: `ml/examples/quick_performance_benchmark.rs` -- **CI Workflow**: `.github/workflows/performance.yml` diff --git a/crates/ml/PERFORMANCE_TRACKING.md b/crates/ml/PERFORMANCE_TRACKING.md deleted file mode 100644 index 6c4e66685..000000000 --- a/crates/ml/PERFORMANCE_TRACKING.md +++ /dev/null @@ -1,261 +0,0 @@ -# Performance Regression Detection System - -Automated performance tracking and regression detection for ML training pipeline. - -## Overview - -This TDD-driven system tracks key performance metrics and automatically fails CI builds when performance degrades beyond acceptable thresholds (>10% regression). - -## Tracked Metrics - -| Metric | Target | Description | -|--------|--------|-------------| -| **DBN Load Time** | <10ms | Real market data loading from DBN files | -| **Feature Extraction** | - | Technical indicator calculation (16 features) | -| **Training Step** | - | Single training iteration time | -| **Inference Latency** | <50ฮผs | Model prediction time (HFT requirement) | -| **Throughput** | - | Samples processed per second | -| **Memory Usage** | Model-specific | Peak memory consumption | - -## Architecture - -``` -โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” -โ”‚ Performance Regression Detection โ”‚ -โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ - โ”‚ - โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” - โ”‚ โ”‚ - โ–ผ โ–ผ -โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” -โ”‚ Baseline โ”‚ โ”‚ Current Run โ”‚ -โ”‚ Metrics โ”‚ โ”‚ Metrics โ”‚ -โ”‚ (saved JSON) โ”‚ โ”‚ (new results) โ”‚ -โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ - โ”‚ โ”‚ - โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ - โ”‚ - โ–ผ - โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” - โ”‚ Regression โ”‚ - โ”‚ Detection โ”‚ - โ”‚ (>10% = FAIL) โ”‚ - โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ - โ”‚ - โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” - โ”‚ โ”‚ - โ–ผ โ–ผ - โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” - โ”‚ CI โ”‚ โ”‚ Grafana โ”‚ - โ”‚ Report โ”‚ โ”‚ Dashboardโ”‚ - โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ -``` - -## Usage - -### 1. Record Baseline - -Run benchmark and save baseline: - -```bash -cargo run --release -p ml --example quick_performance_benchmark -- \ - --output ml/benchmark_results/dqn_baseline.json \ - --git-commit $(git rev-parse HEAD) \ - --model DQN -``` - -### 2. Check for Regression (CI) - -Compare current metrics against baseline: - -```bash -cargo run --release -p ml --example check_performance_regression -- \ - --baseline ml/benchmark_results/dqn_baseline.json \ - --current ml/benchmark_results/dqn_current.json \ - --output regression_report.md \ - --threshold 10.0 -``` - -**Exit Codes:** -- `0` - No regression detected -- `1` - Performance regression detected (>10% degradation) - -### 3. View in Grafana - -Import dashboard: - -```bash -# Copy Grafana dashboard JSON -cp ml/grafana/performance_tracking_dashboard.json /path/to/grafana/dashboards/ - -# Access at: http://localhost:3000 -``` - -## CI Integration - -Performance checks run automatically on every PR: - -```yaml -# .github/workflows/performance.yml -- name: Check for regression - run: | - cargo run --release -p ml --example check_performance_regression -- \ - --baseline baseline.json \ - --current current.json \ - --output report.md - - # Exit code 1 fails the build -``` - -## Testing (TDD Approach) - -All tests pass (12/12): - -```bash -cargo test -p ml --test performance_regression_tests -``` - -Test coverage: -- โœ… Baseline saving/loading -- โœ… Regression detection (>10% threshold) -- โœ… Metric tracking (DBN, features, training, inference) -- โœ… CI integration (exit codes) -- โœ… Multiple models (independent baselines) - -## File Structure - -``` -ml/ -โ”œโ”€โ”€ src/benchmark/ -โ”‚ โ””โ”€โ”€ performance_tracker.rs # Core regression detection -โ”œโ”€โ”€ tests/ -โ”‚ โ””โ”€โ”€ performance_regression_tests.rs # 12 TDD tests -โ”œโ”€โ”€ examples/ -โ”‚ โ”œโ”€โ”€ quick_performance_benchmark.rs # Record metrics -โ”‚ โ””โ”€โ”€ check_performance_regression.rs # Detect regressions -โ”œโ”€โ”€ grafana/ -โ”‚ โ””โ”€โ”€ performance_tracking_dashboard.json # Grafana dashboard -โ””โ”€โ”€ benchmark_results/ - โ”œโ”€โ”€ dqn_baseline.json # DQN baseline - โ”œโ”€โ”€ ppo_baseline.json # PPO baseline - โ”œโ”€โ”€ mamba2_baseline.json # MAMBA-2 baseline - โ””โ”€โ”€ tft_baseline.json # TFT baseline -``` - -## Regression Threshold - -**10% threshold** = Fail CI if any metric degrades by >10% - -Example: -- Baseline: DBN load time = 0.70ms -- Current: DBN load time = 0.80ms (14.3% slower) -- Result: โŒ **FAIL** - Performance regression detected - -## Model-Specific Baselines - -Each model has independent baselines: - -| Model | Memory | Training Step | Inference | -|-------|--------|---------------|-----------| -| DQN | 50-150MB | ~100ms | ~45ฮผs | -| PPO | 50-200MB | ~150ms | ~50ฮผs | -| MAMBA-2 | 150-500MB | ~200ms | ~40ฮผs | -| TFT | 1.5-2.5GB | ~500ms | ~55ฮผs | - -## Example Report - -```markdown -# Performance Regression Check - -## โŒ Regression Detected - -Performance regression detected: 2 metric(s) degraded by >10%: dbn_load_time_ms, training_step_time_ms - -### Regressions - -| Metric | Baseline | Current | Change | -|--------|----------|---------|--------| -| dbn_load_time_ms | 0.70 | 0.81 | +15.7% | -| training_step_time_ms | 100.00 | 120.00 | +20.0% | - -### Details - -- DBN data loading time increased by 15.7% (0.70 โ†’ 0.81) -- Training step time increased by 20.0% (100.00 โ†’ 120.00) - -**Baseline**: 2025-10-15 10:00:00 (commit: abc123) -**Current**: 2025-10-15 10:05:00 (commit: def456) -``` - -## Grafana Dashboard - -Track performance over time: - -- **DBN Load Time** - Real-time monitoring with <10ms threshold -- **Inference Latency** - Per-model tracking (<50ฮผs target) -- **Training Step Time** - Compare models (DQN, PPO, MAMBA-2, TFT) -- **Memory Usage** - Track memory consumption by model -- **Regression Count** - Total regressions detected -- **Performance Change** - % change vs baseline - -## Integration with Existing Systems - -### GPU Training Benchmark - -Performance tracker integrates with existing benchmark system: - -```rust -use ml::benchmark::{PerformanceTracker, PerformanceMetrics}; - -// After benchmark run -let metrics = PerformanceMetrics { - dbn_load_time_ms: benchmark_results.dbn_load_time, - feature_extraction_time_ms: benchmark_results.feature_time, - training_step_time_ms: benchmark_results.training_time, - inference_latency_us: benchmark_results.inference_latency, - throughput_samples_per_sec: benchmark_results.throughput, - memory_usage_mb: benchmark_results.memory_peak, - timestamp: Utc::now(), - git_commit: git_commit_hash, - model_type: "DQN".to_string(), -}; - -tracker.record_metrics(metrics).await?; -tracker.save_baseline().await?; -``` - -### Continuous Monitoring - -Pipeline integration: - -``` -Training Run โ†’ Record Metrics โ†’ Check Regression โ†’ Update Grafana - โ†“ โ†“ โ†“ โ†“ - model.pt baseline.json report.md (CI) Prometheus metrics -``` - -## Benefits - -1. **Automated Detection**: Catch performance regressions before merge -2. **Historical Tracking**: Grafana dashboards show trends over time -3. **CI Integration**: Fail builds on >10% regression -4. **Model-Specific**: Independent baselines per model -5. **TDD Tested**: 12/12 tests passing (100% coverage) - -## Future Enhancements - -- [ ] Statistical significance testing (t-test) -- [ ] Performance budget per model -- [ ] Automatic baseline updates on main merge -- [ ] Slack/email notifications on regression -- [ ] P95/P99 latency tracking -- [ ] GPU utilization metrics -- [ ] Multi-epoch stability analysis - -## References - -- Test file: `ml/tests/performance_regression_tests.rs` -- Implementation: `ml/src/benchmark/performance_tracker.rs` -- CI workflow: `.github/workflows/performance.yml` -- Dashboard: `ml/grafana/performance_tracking_dashboard.json` -- CLAUDE.md: System architecture and targets diff --git a/crates/ml/QUICKSTART_GPU_BENCHMARK.md b/crates/ml/QUICKSTART_GPU_BENCHMARK.md deleted file mode 100644 index 0290fcd1c..000000000 --- a/crates/ml/QUICKSTART_GPU_BENCHMARK.md +++ /dev/null @@ -1,110 +0,0 @@ -# GPU Training Benchmark - Quick Start - -## Prerequisites -- RTX 3050 Ti GPU (4GB VRAM) or equivalent -- CUDA 12.8+ installed -- 360 DBN files in `test_data/real/databento/ml_training/` - -## 1. Run Benchmark (Default: 10 epochs) -```bash -cd /home/jgrusewski/Work/foxhunt -cargo run -p ml --example gpu_training_benchmark --release -``` - -**Expected Duration**: 30-60 minutes -**Output**: `ml/benchmark_results/gpu_training_benchmark_{timestamp}.json` - -## 2. Interpret Results - -The JSON report contains: -- `dqn_results.statistics.mean_seconds` - Average epoch time for DQN -- `ppo_results.statistics.mean_seconds` - Average epoch time for PPO -- `decision.recommendation` - "local_gpu" or "cloud_gpu" -- `decision.rationale` - Why this recommendation - -## 3. Decision Framework - -**Recommendation: "local_gpu"** -- Total training time < 24 hours -- Action: Proceed with local RTX 3050 Ti training -- Cost: $0 (local hardware) - -**Recommendation: "cloud_gpu"** -- Total training time > 48 hours -- Action: Use cloud GPU (A100 recommended) -- Cost: ~$250/week - -**Recommendation: "either"** -- Total training time 24-48 hours -- Action: User choice based on urgency/cost -- Compare `estimated_cost_local` vs `estimated_cost_cloud` - -## 4. Advanced Options - -### Custom epochs (more accurate) -```bash -cargo run -p ml --example gpu_training_benchmark --release -- --epochs 20 -``` - -### CPU-only mode (testing) -```bash -cargo run -p ml --example gpu_training_benchmark --release -- --cpu-only -``` - -### Custom output path -```bash -cargo run -p ml --example gpu_training_benchmark --release -- --output my_results.json -``` - -## 5. Troubleshooting - -**Error: "CUDA out of memory"** -- TFT model requires 1.5-2.5GB VRAM -- Solution: Batch size automatically limited to 4 -- If still OOM: Close other GPU applications - -**Warning: "Thermal throttling detected"** -- GPU temperature >85ยฐC -- Solution: Improve cooling, reduce room temperature -- Training will continue but slower - -**Error: "DBN files not found"** -- Missing training data -- Solution: Run the download pipeline first: - ```bash - cargo run -p ml --example download_baseline --release - ``` - -## 6. Full Documentation - -See `ml/docs/GPU_BENCHMARK_GUIDE.md` for comprehensive documentation. - -## Expected Output Example - -```json -{ - "timestamp": "2025-10-13T12:30:00Z", - "decision": { - "recommendation": "local_gpu", - "rationale": "Training time 18.5 hours < 24h threshold. Local GPU viable.", - "estimated_local_hours": 18.5, - "estimated_cost_local": 0.42, - "estimated_cost_cloud": 9.73 - }, - "dqn_results": { - "statistics": { - "mean_seconds": 0.045, - "confidence_interval_95": [0.042, 0.048], - "p95": 0.052, - "p99": 0.058 - }, - "memory_peak_mb": 127.3, - "stability": {"is_stable": true} - }, - "ppo_results": { ... } -} -``` - ---- - -**Questions?** See full documentation or troubleshooting guide. diff --git a/crates/ml/docs/GPU_BENCHMARK_GUIDE.md b/crates/ml/docs/GPU_BENCHMARK_GUIDE.md deleted file mode 100644 index b0418d663..000000000 --- a/crates/ml/docs/GPU_BENCHMARK_GUIDE.md +++ /dev/null @@ -1,2057 +0,0 @@ -# GPU Training Benchmark System Guide - -**Version**: 1.0 -**Last Updated**: 2025-10-13 -**Target Hardware**: NVIDIA RTX 3050 Ti (4GB VRAM) -**Purpose**: Empirical GPU training time measurement for production decision-making - ---- - -## Table of Contents - -1. [Overview](#overview) -2. [Quick Start Guide](#quick-start-guide) -3. [Architecture Documentation](#architecture-documentation) -4. [Statistical Methodology](#statistical-methodology) -5. [Decision Framework](#decision-framework) -6. [Interpreting Results](#interpreting-results) -7. [Troubleshooting](#troubleshooting) -8. [Advanced Configuration](#advanced-configuration) -9. [Performance Expectations](#performance-expectations) -10. [References](#references) - ---- - -## Overview - -### Purpose - -The GPU Training Benchmark System provides **empirical, statistically rigorous measurements** of machine learning model training times on local GPU hardware. This system eliminates guesswork and projections by running actual training workloads and measuring real performance characteristics. - -### The Problem - -Trading system development requires critical decisions about ML infrastructure investment: - -- **Local GPU Training**: $0 cost, but potentially slow (4-6 weeks commitment?) -- **Cloud GPU Training**: ~$250/week cost, faster but recurring expense -- **No Baseline Data**: Without empirical measurements, teams make expensive decisions based on guesswork - -A wrong decision costs either **time** (slow local training blocking production) or **money** (unnecessary cloud GPU rental). - -### The Solution - -This benchmark system provides a **3-hour investment** to gather **empirical data** that informs a **4-6 week decision**. By running controlled experiments with statistical rigor, you get: - -1. **Actual training times** (not projections) -2. **Memory usage profiles** (will 4GB VRAM suffice?) -3. **Stability validation** (is training numerically sound?) -4. **95% confidence intervals** (how reliable are these numbers?) -5. **Clear recommendations** (local GPU viable or cloud GPU needed?) - -### Architecture Overview - -The system consists of **6 core modules** that orchestrate **4 model benchmarks**: - -**Core Modules**: -1. **GPU Hardware Manager**: Device initialization, thermal monitoring, warmup sequences -2. **Statistical Sampler**: Confidence intervals, percentile analysis, outlier detection -3. **Batch Size Finder**: Binary search for optimal batch size, OOM detection -4. **Memory Profiler**: VRAM tracking via nvidia-smi integration -5. **Stability Validator**: Loss trends, gradient health, NaN detection -6. **Benchmark Coordinator**: Orchestrates all modules, generates reports - -**Model Benchmarks**: -1. **DQN (Deep Q-Network)**: Reinforcement learning for discrete action spaces -2. **PPO (Proximal Policy Optimization)**: Policy gradient RL with stability -3. **MAMBA-2**: State space model for sequence prediction -4. **TFT (Temporal Fusion Transformer)**: Multi-horizon time series forecasting - -### Key Features - -**Statistical Rigor**: -- Minimum 10-20 epoch sampling (not 5!) -- Warmup epoch removal (first 2 epochs discarded) -- Outlier detection via 3-sigma rule -- 95% confidence intervals using t-distribution -- Coefficient of variation analysis (<0.15 for stable measurements) - -**Memory Safety**: -- Binary search for maximum viable batch size -- Gradient accumulation for OOM scenarios -- Real-time VRAM monitoring -- Automatic fallback to smaller batches - -**Production Readiness**: -- Full error handling and recovery -- Structured JSON reports -- Actionable recommendations -- Reproducible results - -### System Requirements - -**Hardware**: -- NVIDIA GPU with 4GB+ VRAM (tested on RTX 3050 Ti) -- CUDA 12.8 or higher -- 16GB+ system RAM recommended - -**Software**: -- Rust 1.70+ with cargo -- CUDA Toolkit installed -- nvidia-smi accessible in PATH - -**Data**: -- 360 DBN (DataBento) files in `test_data/real/databento/ml_training/` -- Total size: ~2-5GB (Parquet format) - -### Expected Runtime - -**Full Benchmark** (10 epochs ร— 4 models): -- Runtime: 30-60 minutes on RTX 3050 Ti -- Output: JSON report with all statistics -- Decision: Clear recommendation on GPU strategy - -**Quick Test** (5 epochs ร— 2 models): -- Runtime: 10-15 minutes -- Output: Preliminary estimates -- Decision: Initial feasibility check - ---- - -## Quick Start Guide - -### Prerequisites - -Before running the benchmark, ensure: - -```bash -# 1. Verify CUDA installation -nvcc --version -# Expected: CUDA 12.8 or higher - -# 2. Check GPU availability -nvidia-smi -# Expected: RTX 3050 Ti with 4GB VRAM visible - -# 3. Verify data files exist -ls -l test_data/real/databento/ml_training/*.dbn.zst | wc -l -# Expected: 360 files - -# 4. Check Rust toolchain -cargo --version -# Expected: cargo 1.70+ -``` - -### Running the Benchmark - -**Standard Full Benchmark** (recommended first run): - -```bash -cd /home/jgrusewski/Work/foxhunt - -# Run with 10 epochs (30-60 min runtime) -cargo run -p ml --example gpu_training_benchmark --release -- --epochs 10 -``` - -**Quick Feasibility Check** (faster, less accurate): - -```bash -# Run with 5 epochs (15-20 min runtime) -cargo run -p ml --example gpu_training_benchmark --release -- --epochs 5 -``` - -**High-Confidence Measurement** (if variance is high): - -```bash -# Run with 20 epochs (60-90 min runtime) -cargo run -p ml --example gpu_training_benchmark --release -- --epochs 20 -``` - -### Expected Output - -The benchmark will display real-time progress: - -``` -[INFO] GPU Benchmark System v1.0 -[INFO] Target Device: NVIDIA GeForce RTX 3050 Ti Laptop GPU -[INFO] VRAM Available: 4096 MB -[INFO] Data files: 360 DBN files detected - -[INFO] === Phase 1: Hardware Warmup (2 epochs) === -[INFO] Warmup Epoch 1/2: 45.2ms -[INFO] Warmup Epoch 2/2: 43.8ms -[INFO] GPU temperature: 68ยฐC (stable) - -[INFO] === Phase 2: DQN Training Benchmark === -[INFO] Finding optimal batch size... -[INFO] Optimal batch size: 512 (VRAM usage: 150 MB) -[INFO] Training epochs: 10 -[INFO] Epoch 1/10: 42.3ms (loss: 0.0234) -[INFO] Epoch 2/10: 41.8ms (loss: 0.0198) -... -[INFO] DQN Results: - - Mean epoch time: 42.1ms ยฑ 2.3ms (95% CI) - - P95 latency: 45.2ms - - Memory usage: 150 MB VRAM - - Training stability: STABLE โœ“ - -[INFO] === Phase 3: PPO Training Benchmark === -... - -[INFO] === Phase 4: MAMBA-2 Training Benchmark === -... - -[INFO] === Phase 5: TFT Training Benchmark === -... - -[INFO] === Benchmark Complete === -[INFO] Report saved: ml/benchmark_results/report_20251013_143022.json -[INFO] Recommendation: LOCAL_GPU_VIABLE -[INFO] Estimated full training time: 18.5 hours (DQN + PPO combined) -``` - -### Interpreting the Recommendation - -The system provides one of three recommendations: - -**LOCAL_GPU_VIABLE** โœ…: -- DQN + PPO combined training time <24 hours -- Your RTX 3050 Ti can handle production workloads -- Proceed with local GPU training - -**CLOUD_GPU_RECOMMENDED** โ˜๏ธ: -- DQN + PPO combined training time >48 hours -- Cloud GPU (A100) will be more efficient -- Budget ~$250/week for cloud resources - -**GRAY_ZONE** โš ๏ธ: -- Training time 24-48 hours -- Decision depends on urgency and cost sensitivity -- Consider user preference and hardware availability - -### Accessing Detailed Results - -The JSON report contains full statistical analysis: - -```bash -# View the report -cat ml/benchmark_results/report_20251013_143022.json | jq . - -# Key sections in the report -{ - "timestamp": "2025-10-13T14:30:22Z", - "hardware": { ... }, // GPU specs, driver versions - "models": { - "dqn": { - "mean_epoch_time_ms": 42.1, - "confidence_interval_ms": [39.8, 44.4], - "memory_usage_mb": 150, - "stability": "STABLE" - }, - ... - }, - "recommendation": { - "decision": "LOCAL_GPU_VIABLE", - "reasoning": "Combined DQN+PPO training time: 18.5h < 24h threshold", - "estimated_full_training_days": 31.2 - } -} -``` - -### Next Steps - -Based on the recommendation: - -**If LOCAL_GPU_VIABLE**: -1. Configure ML training pipeline with measured batch sizes -2. Set up overnight training schedule (18-24 hour runs) -3. Monitor first production training run -4. Adjust hyperparameters if needed - -**If CLOUD_GPU_RECOMMENDED**: -1. Provision cloud GPU (AWS p3.2xlarge or equivalent) -2. Estimate monthly costs: ~$1,000/month for continuous training -3. Set up cloud training pipeline -4. Re-run benchmark on cloud hardware to validate - -**If GRAY_ZONE**: -1. Consider urgency: Is 30-40 hours acceptable? -2. Evaluate costs: Is $250/week justifiable? -3. Run extended benchmark (20 epochs) for higher confidence -4. Make informed decision based on team priorities - ---- - -## Architecture Documentation - -### Module 1: GPU Hardware Manager - -**Purpose**: Ensures GPU is in stable, warmed-up state before measurements begin. - -**Rationale**: Cold start measurements can be 2-5x slower than steady-state due to: -- GPU frequency scaling (power saving modes) -- Driver initialization overhead -- CUDA context creation latency -- Thermal throttling during ramp-up - -**Key Algorithms**: - -```rust -// Warmup sequence (2 epochs minimum) -pub fn warmup_gpu(device: &Device, epochs: usize) -> Result { - let mut times = Vec::new(); - - for epoch in 0..epochs { - let start = Instant::now(); - - // Run dummy workload (matrix multiplication) - let a = Tensor::rand(0.0, 1.0, &[1024, 1024], device)?; - let b = Tensor::rand(0.0, 1.0, &[1024, 1024], device)?; - let _c = a.matmul(&b)?; - - device.synchronize()?; // Ensure completion - times.push(start.elapsed()); - } - - Ok(WarmupStats { - mean_time: times.iter().sum::() / times.len(), - temperature: read_gpu_temperature()?, - }) -} -``` - -**Thermal Monitoring**: - -The module monitors GPU temperature via nvidia-smi to detect thermal throttling: - -```rust -pub fn check_thermal_stability() -> Result { - let output = Command::new("nvidia-smi") - .args(&["--query-gpu=temperature.gpu", "--format=csv,noheader"]) - .output()?; - - let temp: f32 = String::from_utf8(output.stdout)?.trim().parse()?; - - match temp { - t if t < 70.0 => Ok(ThermalStatus::Optimal), - t if t < 80.0 => Ok(ThermalStatus::Acceptable), - t if t < 90.0 => Ok(ThermalStatus::Warning), - _ => Err(CommonError::internal("GPU overheating")) - } -} -``` - -**Integration Points**: -- Called once at benchmark start -- Re-checked between model benchmarks -- Aborts if temperature >90ยฐC - -**Configuration Options**: -- `warmup_epochs`: Default 2, range 1-5 -- `thermal_threshold_celsius`: Default 90.0 - ---- - -### Module 2: Statistical Sampler - -**Purpose**: Provides rigorous statistical analysis of training time measurements with confidence intervals and variance assessment. - -**Rationale**: Single-epoch measurements are unreliable due to: -- OS scheduling variance (10-30% variation) -- GPU clock frequency adjustments -- Background process interference -- Thermal throttling fluctuations - -Statistical sampling with 10-20 epochs provides: -- 95% confidence intervals (margin of error) -- Outlier detection (3-sigma rule) -- Variance stability (coefficient of variation) - -**Key Algorithms**: - -**Confidence Interval Calculation** (t-distribution): - -```rust -pub fn compute_confidence_interval( - samples: &[f64], - confidence_level: f64 // 0.95 for 95% CI -) -> (f64, f64) { - let n = samples.len() as f64; - let mean = samples.iter().sum::() / n; - - // Sample standard deviation - let variance = samples.iter() - .map(|x| (x - mean).powi(2)) - .sum::() / (n - 1.0); - let std_dev = variance.sqrt(); - - // t-value for 95% CI with (n-1) degrees of freedom - let df = n - 1.0; - let t_value = student_t_inverse_cdf(1.0 - (1.0 - confidence_level) / 2.0, df); - - // Margin of error - let margin = t_value * std_dev / n.sqrt(); - - (mean - margin, mean + margin) -} -``` - -**Outlier Detection** (3-sigma rule): - -```rust -pub fn remove_outliers(samples: &[f64]) -> Vec { - let mean = samples.iter().sum::() / samples.len() as f64; - let std_dev = compute_std_dev(samples, mean); - - samples.iter() - .filter(|&&x| { - let z_score = (x - mean).abs() / std_dev; - z_score < 3.0 // Keep samples within 3 standard deviations - }) - .copied() - .collect() -} -``` - -**Coefficient of Variation** (stability metric): - -```rust -pub fn compute_cv(samples: &[f64]) -> f64 { - let mean = samples.iter().sum::() / samples.len() as f64; - let std_dev = compute_std_dev(samples, mean); - - std_dev / mean // Dimensionless measure of variability -} - -// Stability assessment -pub fn is_stable(cv: f64) -> bool { - cv < 0.15 // <15% variation considered stable -} -``` - -**Percentile Calculation** (P95/P99 latencies): - -```rust -pub fn compute_percentile(samples: &[f64], p: f64) -> f64 { - let mut sorted = samples.to_vec(); - sorted.sort_by(|a, b| a.partial_cmp(b).unwrap()); - - let index = ((p / 100.0) * sorted.len() as f64) as usize; - sorted[index.min(sorted.len() - 1)] -} -``` - -**Integration Points**: -- Called after each model's training epochs -- Outliers removed before final statistics -- CV checked to ensure measurement stability - -**Configuration Options**: -- `confidence_level`: Default 0.95 (95% CI) -- `outlier_sigma`: Default 3.0 -- `cv_threshold`: Default 0.15 - ---- - -### Module 3: Batch Size Finder - -**Purpose**: Determines the maximum viable batch size that fits in GPU memory without triggering out-of-memory (OOM) errors. - -**Rationale**: Larger batch sizes improve GPU utilization but risk OOM. The optimal batch size: -- Maximizes throughput (GPU compute saturation) -- Avoids memory fragmentation -- Leaves headroom for gradient computation - -**Key Algorithm** (Binary Search): - -```rust -pub fn find_optimal_batch_size( - model: &dyn TrainableModel, - device: &Device, - initial_batch: usize, // Start at 512 - max_batch: usize // Cap at 4096 -) -> Result { - let mut low = 32; - let mut high = max_batch; - let mut optimal = initial_batch; - - while low <= high { - let mid = (low + high) / 2; - - match test_batch_size(model, device, mid) { - Ok(vram_used) => { - // Success - try larger batch - optimal = mid; - low = mid + 1; - - // Stop if VRAM usage >90% (headroom for gradients) - if vram_used > device.total_memory() * 0.9 { - break; - } - } - Err(e) if is_oom_error(&e) => { - // OOM - try smaller batch - high = mid - 1; - } - Err(e) => return Err(e), // Other error - } - } - - // Verify optimal batch works reliably (3 attempts) - for _ in 0..3 { - test_batch_size(model, device, optimal)?; - } - - Ok(BatchSizeConfig { - batch_size: optimal, - gradient_accumulation_steps: compute_accumulation_steps(optimal), - estimated_vram_mb: measure_vram_usage(model, device, optimal)?, - }) -} -``` - -**Gradient Accumulation** (for OOM scenarios): - -If the optimal batch size is too small (<128), gradient accumulation simulates larger effective batch sizes: - -```rust -pub fn compute_accumulation_steps(batch_size: usize) -> usize { - let target_effective_batch = 512; - - if batch_size >= target_effective_batch { - 1 // No accumulation needed - } else { - target_effective_batch / batch_size // Accumulate to reach target - } -} -``` - -**OOM Detection**: - -```rust -pub fn is_oom_error(error: &CommonError) -> bool { - let msg = error.to_string().to_lowercase(); - msg.contains("out of memory") || - msg.contains("oom") || - msg.contains("cuda error 2") // CUDA_ERROR_OUT_OF_MEMORY -} -``` - -**Integration Points**: -- Called once per model before training -- Results cached and reused across epochs -- Gradients monitored during training to ensure stability - -**Configuration Options**: -- `initial_batch_size`: Default 512 -- `max_batch_size`: Default 4096 -- `target_effective_batch`: Default 512 - ---- - -### Module 4: Memory Profiler - -**Purpose**: Tracks GPU memory usage throughout training to detect memory leaks and validate batch size configurations. - -**Rationale**: Memory issues manifest as: -- Gradual VRAM accumulation (memory leaks) -- Sudden spikes (inefficient operations) -- OOM crashes (batch size too large) - -Real-time monitoring via nvidia-smi provides: -- Current VRAM usage -- Peak VRAM usage -- Memory fragmentation indicators - -**Key Algorithm** (nvidia-smi Integration): - -```rust -pub struct MemorySnapshot { - pub timestamp: Instant, - pub used_mb: u64, - pub free_mb: u64, - pub total_mb: u64, - pub utilization_percent: f32, -} - -pub fn capture_memory_snapshot() -> Result { - let output = Command::new("nvidia-smi") - .args(&[ - "--query-gpu=memory.used,memory.free,memory.total", - "--format=csv,noheader,nounits" - ]) - .output()?; - - let line = String::from_utf8(output.stdout)?; - let parts: Vec<&str> = line.trim().split(',').collect(); - - let used_mb: u64 = parts[0].trim().parse()?; - let free_mb: u64 = parts[1].trim().parse()?; - let total_mb: u64 = parts[2].trim().parse()?; - - Ok(MemorySnapshot { - timestamp: Instant::now(), - used_mb, - free_mb, - total_mb, - utilization_percent: (used_mb as f32 / total_mb as f32) * 100.0, - }) -} -``` - -**Memory Leak Detection**: - -```rust -pub fn detect_memory_leak(snapshots: &[MemorySnapshot]) -> Option { - if snapshots.len() < 5 { - return None; // Need multiple samples - } - - // Linear regression on memory usage over time - let slope = compute_memory_trend_slope(snapshots); - - // Leak detected if memory grows >5 MB/epoch consistently - if slope > 5.0 { - Some(LeakReport { - growth_rate_mb_per_epoch: slope, - estimated_oom_in_epochs: (4096.0 - snapshots.last().unwrap().used_mb as f32) / slope, - }) - } else { - None - } -} -``` - -**Integration Points**: -- Snapshot captured before/after each epoch -- Leak detection runs every 5 epochs -- Alerts if memory growth detected - -**Configuration Options**: -- `snapshot_interval_epochs`: Default 1 -- `leak_threshold_mb_per_epoch`: Default 5.0 - ---- - -### Module 5: Stability Validator - -**Purpose**: Ensures training is numerically stable and converging properly by monitoring loss trends, gradient magnitudes, and detecting NaN/Inf values. - -**Rationale**: Unstable training manifests as: -- Exploding gradients (>1000 magnitude) -- Vanishing gradients (<1e-7 magnitude) -- NaN/Inf loss values -- Non-decreasing loss over time - -Stability validation prevents: -- Wasted training time on divergent runs -- Misleading benchmark results from unstable training - -**Key Algorithms**: - -**Loss Trend Analysis**: - -```rust -pub fn analyze_loss_trend(loss_history: &[f32]) -> TrendAnalysis { - if loss_history.len() < 5 { - return TrendAnalysis::Insufficient; - } - - // Linear regression on recent losses - let recent = &loss_history[loss_history.len() - 5..]; - let slope = compute_slope(recent); - - match slope { - s if s < -0.01 => TrendAnalysis::Decreasing, // Healthy - s if s.abs() <= 0.01 => TrendAnalysis::Stable, // Plateau (acceptable) - _ => TrendAnalysis::Increasing, // Divergence (bad) - } -} -``` - -**Gradient Health Check**: - -```rust -pub fn check_gradient_health(gradients: &[Tensor]) -> GradientStatus { - let mut max_magnitude = 0.0; - let mut min_magnitude = f32::INFINITY; - - for grad in gradients { - // Check for NaN/Inf - if has_nan_or_inf(grad) { - return GradientStatus::Invalid; - } - - // Compute gradient magnitude - let magnitude = grad.abs()?.max()?.to_scalar::()?; - max_magnitude = max_magnitude.max(magnitude); - min_magnitude = min_magnitude.min(magnitude); - } - - match (max_magnitude, min_magnitude) { - (max, _) if max > 1000.0 => GradientStatus::Exploding, - (_, min) if min < 1e-7 => GradientStatus::Vanishing, - _ => GradientStatus::Healthy, - } -} -``` - -**NaN/Inf Detection**: - -```rust -pub fn has_nan_or_inf(tensor: &Tensor) -> bool { - let values = tensor.flatten_all()?.to_vec1::()?; - values.iter().any(|&v| !v.is_finite()) -} -``` - -**Integration Points**: -- Loss checked after every epoch -- Gradients checked every 5 epochs -- Training aborted if instability detected - -**Configuration Options**: -- `gradient_check_interval`: Default 5 epochs -- `exploding_threshold`: Default 1000.0 -- `vanishing_threshold`: Default 1e-7 - ---- - -### Module 6: Benchmark Coordinator - -**Purpose**: Orchestrates all modules, executes model benchmarks in sequence, and generates comprehensive JSON reports. - -**Workflow**: - -```rust -pub async fn run_benchmark(config: BenchmarkConfig) -> Result { - // Phase 1: Hardware initialization - let hardware_info = gpu_hardware_manager::initialize()?; - gpu_hardware_manager::warmup_gpu(&hardware_info.device, 2)?; - - // Phase 2: Run model benchmarks - let mut model_results = HashMap::new(); - - for model_type in [ModelType::DQN, ModelType::PPO, ModelType::MAMBA2, ModelType::TFT] { - log::info!("Benchmarking {}", model_type); - - // Find optimal batch size - let batch_config = batch_size_finder::find_optimal_batch_size( - model_type, - &hardware_info.device, - 512, - 4096 - )?; - - // Run training epochs - let mut epoch_times = Vec::new(); - let mut loss_history = Vec::new(); - - for epoch in 0..config.epochs { - let start = Instant::now(); - let loss = train_single_epoch(model_type, &batch_config)?; - let elapsed = start.elapsed(); - - epoch_times.push(elapsed.as_secs_f64() * 1000.0); // Convert to ms - loss_history.push(loss); - - // Capture memory snapshot - let memory = memory_profiler::capture_memory_snapshot()?; - - // Validate stability - if epoch % 5 == 4 { - stability_validator::check_gradient_health(...)?; - } - } - - // Statistical analysis - let filtered_times = statistical_sampler::remove_outliers(&epoch_times[2..]); // Skip warmup epochs - let (ci_low, ci_high) = statistical_sampler::compute_confidence_interval(&filtered_times, 0.95); - let mean = filtered_times.iter().sum::() / filtered_times.len() as f64; - let p95 = statistical_sampler::compute_percentile(&filtered_times, 95.0); - let p99 = statistical_sampler::compute_percentile(&filtered_times, 99.0); - let cv = statistical_sampler::compute_cv(&filtered_times); - - model_results.insert(model_type, ModelBenchmarkResult { - mean_epoch_time_ms: mean, - confidence_interval_ms: (ci_low, ci_high), - p95_latency_ms: p95, - p99_latency_ms: p99, - coefficient_of_variation: cv, - memory_usage_mb: batch_config.estimated_vram_mb, - stability: stability_validator::analyze_loss_trend(&loss_history), - }); - } - - // Phase 3: Generate recommendation - let recommendation = generate_recommendation(&model_results)?; - - // Phase 4: Save report - let report = BenchmarkReport { - timestamp: Utc::now(), - hardware: hardware_info, - models: model_results, - recommendation, - }; - - save_report(&report)?; - Ok(report) -} -``` - -**Integration Points**: -- Entry point for entire benchmark system -- Calls all other modules in correct sequence -- Handles errors and generates reports - -**Configuration Options**: -- `epochs`: Default 10, range 5-20 -- `output_dir`: Default `ml/benchmark_results/` - ---- - -## Statistical Methodology - -### Why 10-20 Epochs Minimum? - -**Problem**: Single-epoch measurements are unreliable. - -**Example**: Measuring epoch time once might yield 50ms, but the true mean could be 45ms or 55ms (ยฑ10% variance). - -**Solution**: Sample multiple epochs and apply statistical inference. - -With 10 epochs: -- 95% confidence interval narrows to ยฑ5% (margin of error) -- Outliers can be detected and removed -- Variance stability can be assessed - -With 20 epochs: -- 95% confidence interval narrows to ยฑ3% -- Higher confidence in estimates -- Better outlier detection - -**Statistical Power**: - -``` -Sample Size | Margin of Error (95% CI) | Confidence -------------|--------------------------|------------ -1 epoch | Unknown | 0% -5 epochs | ยฑ15% | Low -10 epochs | ยฑ5% | Moderate -20 epochs | ยฑ3% | High -``` - -### Warmup Epoch Removal - -**Problem**: First 1-2 epochs are significantly slower than steady-state. - -**Causes**: -- CUDA kernel compilation (JIT) -- GPU frequency ramping (power management) -- Driver initialization overhead -- Cache cold misses - -**Solution**: Discard first 2 epochs from statistical analysis. - -**Example**: - -``` -Epoch 1: 150ms (cold start) -Epoch 2: 80ms (ramping) -Epoch 3: 45ms (steady-state) -Epoch 4: 43ms -Epoch 5: 44ms -... - -Mean (all epochs): 72ms โŒ Inaccurate (includes cold start) -Mean (epochs 3+): 44ms โœ… Accurate (steady-state only) -``` - -### Outlier Detection (3-Sigma Rule) - -**Problem**: Occasional outliers skew statistics (OS scheduling, thermal throttling). - -**Solution**: Remove data points >3 standard deviations from the mean. - -**Algorithm**: - -``` -1. Compute mean ฮผ and standard deviation ฯƒ -2. For each sample x: - z-score = |x - ฮผ| / ฯƒ - if z-score > 3: remove sample -3. Recompute statistics on filtered data -``` - -**Example**: - -``` -Epoch times: [42, 43, 44, 41, 150, 43, 42, 44] - ^^^ - outlier (thermal throttle) - -Mean: 56.1ms โŒ Skewed by outlier -Filtered mean: 42.7ms โœ… Outlier removed -``` - -### 95% Confidence Intervals - -**Purpose**: Quantify uncertainty in mean epoch time estimate. - -**Interpretation**: "We are 95% confident the true mean lies within this interval." - -**Formula** (t-distribution): - -``` -CI = [ฮผ - t * (ฯƒ / โˆšn), ฮผ + t * (ฯƒ / โˆšn)] - -where: - ฮผ = sample mean - ฯƒ = sample standard deviation - n = sample size - t = t-value for 95% confidence with (n-1) degrees of freedom -``` - -**Example**: - -``` -10 epochs: [42, 43, 44, 41, 43, 42, 44, 43, 42, 43] ms -Mean: 42.7ms -Std dev: 0.95ms -95% CI: [42.0ms, 43.4ms] - -Interpretation: True mean is likely 42.0-43.4ms (ยฑ0.7ms margin) -``` - -### Coefficient of Variation - -**Purpose**: Assess measurement stability (dimensionless variability metric). - -**Formula**: - -``` -CV = ฯƒ / ฮผ - -where: - ฯƒ = standard deviation - ฮผ = mean -``` - -**Thresholds**: - -``` -CV < 0.10 (10%): Excellent stability โœ… -CV < 0.15 (15%): Acceptable stability โœ… -CV < 0.25 (25%): Marginal stability โš ๏ธ -CV > 0.25 (25%): Poor stability โŒ (need more epochs) -``` - -**Example**: - -``` -Model A: Mean=42ms, StdDev=2ms โ†’ CV=0.048 (4.8%) โœ… Stable -Model B: Mean=200ms, StdDev=50ms โ†’ CV=0.25 (25%) โš ๏ธ Marginal -``` - -### Percentile Analysis (P95/P99) - -**Purpose**: Capture tail latencies (worst-case performance). - -**Rationale**: Mean doesn't reflect worst-case scenarios important for SLAs. - -**Definitions**: -- **P95 (95th percentile)**: 95% of epochs are faster than this -- **P99 (99th percentile)**: 99% of epochs are faster than this - -**Example**: - -``` -100 epochs sorted by time: -[41, 41, 42, 42, 42, 43, 43, 43, 44, 44, ..., 50, 51, 55, 60] - -Mean: 43ms -P95: 50ms (95th sample out of 100) -P99: 60ms (99th sample out of 100) - -Interpretation: Typical epoch is 43ms, but worst-case can reach 60ms -``` - -**Usage**: If P99 >> Mean, investigate variance sources (thermal throttling, background processes). - ---- - -## Decision Framework - -### Overview - -The benchmark system provides an automated recommendation based on empirical measurements. The decision optimizes for **cost-effectiveness** while ensuring **production viability**. - -### Decision Criteria - -**Metric**: Combined DQN + PPO training time for 90-day dataset. - -**Rationale**: -- DQN and PPO are the primary production models (RL agents) -- MAMBA-2 and TFT are optional (forecasting augmentation) -- Focus on critical path for deployment - -**Calculation**: - -``` -Total Training Time = (DQN Mean Epoch Time ร— DQN Total Epochs) + - (PPO Mean Epoch Time ร— PPO Total Epochs) - -where: - DQN Total Epochs = 90 days ร— 288 epochs/day (5-min intervals) - PPO Total Epochs = 90 days ร— 288 epochs/day -``` - -### Recommendation: LOCAL_GPU_VIABLE - -**Condition**: Total training time <24 hours - -**Rationale**: -- Overnight training is acceptable (set-and-forget) -- Zero infrastructure cost (local hardware) -- No cloud egress/ingress latency - -**Action Items**: -1. Configure ML training pipeline with measured batch sizes -2. Schedule training runs during off-hours (8pm-8am) -3. Monitor first production run for stability -4. Set up automated checkpointing (every 2 hours) - -**Cost Analysis**: -``` -Local GPU: $0/month (hardware already owned) -Cloud GPU: $1,000/month (24/7 training) -Savings: $12,000/year -``` - -**Example Output**: - -```json -{ - "decision": "LOCAL_GPU_VIABLE", - "reasoning": "Combined DQN+PPO training time: 18.5h < 24h threshold", - "estimated_full_training_hours": 18.5, - "estimated_full_training_days": 0.77, - "cost_analysis": { - "local_gpu_monthly_cost_usd": 0, - "cloud_gpu_monthly_cost_usd": 1000, - "annual_savings_usd": 12000 - }, - "recommended_actions": [ - "Configure batch sizes: DQN=512, PPO=256", - "Schedule overnight training runs", - "Enable checkpointing every 2 hours", - "Monitor GPU temperature (<80ยฐC)" - ] -} -``` - -### Recommendation: CLOUD_GPU_RECOMMENDED - -**Condition**: Total training time >48 hours - -**Rationale**: -- >2 days per training run is impractical -- Cloud GPU (A100) is 5-10x faster (80GB VRAM, higher compute) -- $250/week is justified by velocity gains - -**Action Items**: -1. Provision AWS p3.2xlarge (V100 16GB) or p4d (A100 40GB) -2. Estimate monthly cost: ~$1,000 for continuous training -3. Set up cloud training pipeline with S3 data transfer -4. Re-run benchmark on cloud hardware to validate speedup - -**Cost Analysis**: - -``` -Local GPU: $0/month + 60 hours/training (unacceptable) -Cloud GPU: $1,000/month + 6-12 hours/training (acceptable) -Time saved: 48 hours/training -``` - -**Example Output**: - -```json -{ - "decision": "CLOUD_GPU_RECOMMENDED", - "reasoning": "Combined DQN+PPO training time: 65h > 48h threshold. Local GPU too slow.", - "estimated_full_training_hours": 65.0, - "estimated_full_training_days": 2.7, - "cost_analysis": { - "local_gpu_monthly_cost_usd": 0, - "cloud_gpu_monthly_cost_usd": 1000, - "cloud_speedup_factor": 8.5, - "cloud_training_hours": 7.6 - }, - "recommended_actions": [ - "Provision AWS p3.2xlarge (V100 16GB VRAM)", - "Budget $1,000/month for continuous training", - "Set up S3 data pipeline for cloud transfer", - "Re-benchmark on cloud GPU to validate 8x speedup" - ] -} -``` - -### Recommendation: GRAY_ZONE - -**Condition**: Total training time 24-48 hours - -**Rationale**: No clear winnerโ€”depends on priorities. - -**Trade-offs**: - -| Factor | Local GPU (24-48h) | Cloud GPU (3-6h) | -|----------------------|--------------------|-------------------| -| Cost | $0/month โœ… | $1,000/month โŒ | -| Training time | 1-2 days โš ๏ธ | 3-6 hours โœ… | -| Iteration velocity | Slow (days) โŒ | Fast (hours) โœ… | -| Hardware utilization | High โœ… | Low (idle cost) โŒ| - -**Decision Factors**: - -1. **Urgency**: Is time-to-market critical? โ†’ Cloud GPU -2. **Budget**: Is $12K/year acceptable? โ†’ Cloud GPU -3. **Experimentation**: Frequent retraining? โ†’ Cloud GPU -4. **Stability**: Mature strategy, rare retraining? โ†’ Local GPU - -**Example Output**: - -```json -{ - "decision": "GRAY_ZONE", - "reasoning": "Combined DQN+PPO training time: 36h (between 24h-48h thresholds). Decision depends on priorities.", - "estimated_full_training_hours": 36.0, - "estimated_full_training_days": 1.5, - "cost_analysis": { - "local_gpu_monthly_cost_usd": 0, - "cloud_gpu_monthly_cost_usd": 1000, - "cloud_speedup_factor": 6.0, - "cloud_training_hours": 6.0 - }, - "decision_factors": { - "urgency": "Is time-to-market critical? If yes โ†’ Cloud GPU", - "budget": "Is $1,000/month acceptable? If yes โ†’ Cloud GPU", - "experimentation": "Frequent model retraining? If yes โ†’ Cloud GPU", - "stability": "Mature strategy, rare updates? If yes โ†’ Local GPU" - }, - "recommended_actions": [ - "Evaluate business urgency and budget constraints", - "Consider hybrid: prototype locally, production on cloud", - "Run extended benchmark (20 epochs) for higher confidence", - "Consult team on priorities: cost vs. velocity" - ] -} -``` - ---- - -## Interpreting Results - -### Understanding the JSON Report - -The benchmark generates a comprehensive JSON report with the following structure: - -```json -{ - "timestamp": "2025-10-13T14:30:22Z", - "hardware": { ... }, - "models": { - "dqn": { ... }, - "ppo": { ... }, - "mamba2": { ... }, - "tft": { ... } - }, - "recommendation": { ... } -} -``` - -### Hardware Section - -```json -"hardware": { - "device_name": "NVIDIA GeForce RTX 3050 Ti Laptop GPU", - "total_vram_mb": 4096, - "cuda_version": "12.8", - "driver_version": "570.86.16", - "compute_capability": "8.6", - "warmup_stats": { - "epochs": 2, - "mean_time_ms": 44.5, - "final_temperature_celsius": 68 - } -} -``` - -**Interpretation**: -- **device_name**: Your GPU model (should match expected hardware) -- **total_vram_mb**: Total GPU memory (4096 MB = 4GB for RTX 3050 Ti) -- **compute_capability**: 8.6 = Ampere architecture (current generation) -- **final_temperature_celsius**: Should be <70ยฐC after warmup (stable thermal state) - -**Red Flags**: -- โš ๏ธ Temperature >75ยฐC: Possible thermal throttling during measurements -- โš ๏ธ VRAM <4096 MB: GPU memory not fully detected -- โš ๏ธ Old driver (<535.x): Update NVIDIA drivers - -### Model Results Section - -```json -"dqn": { - "mean_epoch_time_ms": 42.1, - "confidence_interval_ms": [39.8, 44.4], - "p95_latency_ms": 45.2, - "p99_latency_ms": 47.8, - "coefficient_of_variation": 0.068, - "memory_usage_mb": 150, - "batch_size": 512, - "gradient_accumulation_steps": 1, - "stability": "STABLE", - "loss_trend": "DECREASING" -} -``` - -**Interpretation**: - -**Mean Epoch Time** (42.1ms): -- Primary metric for extrapolation -- Lower is better -- Compare across models to identify bottlenecks - -**Confidence Interval** ([39.8, 44.4] ms): -- True mean likely within this range (95% confidence) -- Narrow interval = high confidence -- Wide interval = high variance (run more epochs) - -**P95/P99 Latency** (45.2ms / 47.8ms): -- Worst-case performance -- If P99 >> Mean: High variance (investigate thermal or background processes) -- If P99 โ‰ˆ Mean: Stable performance โœ… - -**Coefficient of Variation** (0.068 = 6.8%): -- Dimensionless stability metric -- <0.10 (10%): Excellent โœ… -- <0.15 (15%): Acceptable โœ… -- >0.25 (25%): Poor โŒ (need more epochs) - -**Memory Usage** (150 MB): -- VRAM consumed during training -- Sum all models to check if <4GB total -- Leave headroom for OS overhead (~500MB) - -**Batch Size** (512): -- Optimal batch that fits in memory -- Larger = better GPU utilization -- Smaller = risk of underutilization - -**Stability** (STABLE): -- "STABLE": Training converging properly โœ… -- "UNSTABLE": Exploding/vanishing gradients โŒ -- "DIVERGING": Loss increasing (bad hyperparameters) โŒ - -**Loss Trend** (DECREASING): -- "DECREASING": Model learning (expected) โœ… -- "STABLE": Loss plateau (acceptable for later epochs) -- "INCREASING": Divergence (fix learning rate) โŒ - -### Recommendation Section - -```json -"recommendation": { - "decision": "LOCAL_GPU_VIABLE", - "reasoning": "Combined DQN+PPO training time: 18.5h < 24h threshold", - "estimated_full_training_hours": 18.5, - "estimated_full_training_days": 0.77, - "cost_analysis": { - "local_gpu_monthly_cost_usd": 0, - "cloud_gpu_monthly_cost_usd": 1000, - "annual_savings_usd": 12000 - }, - "recommended_actions": [ - "Configure batch sizes: DQN=512, PPO=256", - "Schedule overnight training runs", - "Enable checkpointing every 2 hours" - ] -} -``` - -**Interpretation**: - -**Decision** ("LOCAL_GPU_VIABLE"): -- Clear recommendation based on thresholds -- Follow recommended actions for deployment - -**Estimated Full Training** (18.5 hours): -- Extrapolation to 90-day dataset (25,920 epochs) -- Based on mean epoch time ร— total epochs -- Assumes stable performance (no degradation) - -**Cost Analysis**: -- Annual savings = $12K by avoiding cloud GPU -- Factor into infrastructure ROI calculations - -**Recommended Actions**: -- Concrete next steps for deployment -- Batch sizes validated by benchmark -- Checkpointing strategy to prevent data loss - -### Memory Usage Assessment - -**Total VRAM Calculation**: - -``` -Model | VRAM (MB) | % of 4GB --------------|-----------|---------- -DQN | 150 | 3.7% -PPO | 200 | 4.9% -MAMBA-2 | 400 | 9.8% -TFT | 2400 | 58.5% --------------|-----------|---------- -Total | 3150 | 76.8% โœ… Fits in 4GB with headroom -``` - -**Headroom Check**: -- Total <3500 MB (85%): Safe โœ… -- Total 3500-3800 MB (85-93%): Marginal โš ๏ธ -- Total >3800 MB (>93%): Risk of OOM โŒ - -**Red Flags**: -- โš ๏ธ Any model >2GB individually: May hit fragmentation limits -- โš ๏ธ Total >3800 MB: Reduce batch sizes or use gradient accumulation - -### Extrapolating to Full Training - -**Formula**: - -``` -Full Training Time = (Mean Epoch Time ร— Total Epochs) / 3600 - -where: - Mean Epoch Time = From benchmark (seconds) - Total Epochs = Dataset size / Batch size - 3600 = Seconds per hour -``` - -**Example**: - -``` -DQN Benchmark: - Mean epoch time: 42.1ms = 0.0421s - Benchmark epochs: 10 - -Full Training: - Total epochs: 90 days ร— 288 (5-min bars) = 25,920 epochs - Estimated time: 0.0421s ร— 25,920 = 1,090s = 18.2 minutes - -Wait, that seems too fast! Recheck calculation... - -Actually, the benchmark uses 360 DBN files (days), not epochs. -Each file requires multiple training passes (epochs). - -Correct calculation: - Files: 360 - Epochs per file: 100 (typical for convergence) - Total epochs: 36,000 - Estimated time: 0.0421s ร— 36,000 = 1,515s = 25.3 minutes - -Still fast! DQN is efficient. PPO will take longer. -``` - -**Important**: Verify epoch vs. file semantics in benchmark code before trusting extrapolation. - ---- - -## Troubleshooting - -### Issue: "CUDA out of memory" Error - -**Symptoms**: -``` -Error: CUDA error 2: out of memory -Thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: ...' -``` - -**Causes**: -1. Batch size too large -2. Model architecture too big for 4GB VRAM -3. Memory leak accumulating over epochs -4. Other processes using GPU - -**Solutions**: - -**1. Reduce batch size manually**: -```bash -# Default batch size is 512, try 256 -cargo run -p ml --example gpu_training_benchmark --release -- \ - --epochs 10 \ - --max-batch-size 256 -``` - -**2. Enable gradient accumulation**: -```rust -// In batch_size_finder.rs -let config = BatchSizeConfig { - batch_size: 128, // Smaller batch - gradient_accumulation_steps: 4, // Accumulate 4 batches = effective batch 512 - ... -}; -``` - -**3. Check for memory leaks**: -```bash -# Monitor VRAM usage over time -watch -n 1 nvidia-smi - -# Look for gradual increase (leak indicator) -# Stable VRAM = no leak -``` - -**4. Kill other GPU processes**: -```bash -# List GPU processes -nvidia-smi - -# Kill process using GPU (e.g., PID 12345) -kill -9 12345 -``` - -**5. Simplify model architecture** (last resort): -```rust -// Reduce hidden layers or units -// In ml/src/models/dqn.rs -let hidden_dim = 128; // Was 256 -let num_layers = 2; // Was 4 -``` - ---- - -### Issue: "Training unstable" Warning - -**Symptoms**: -``` -[WARN] Training unstable: loss increasing -[WARN] Gradient health: EXPLODING (magnitude: 2345.67) -``` - -**Causes**: -1. Learning rate too high -2. Gradient clipping disabled -3. Batch size too small (high variance) -4. Initialization issues - -**Solutions**: - -**1. Reduce learning rate**: -```rust -// In model configuration -let learning_rate = 1e-4; // Was 1e-3 (10x reduction) -``` - -**2. Enable gradient clipping**: -```rust -// In training loop -let grad_norm = compute_gradient_norm(&gradients)?; -if grad_norm > 10.0 { - scale_gradients(&mut gradients, 10.0 / grad_norm)?; -} -``` - -**3. Increase batch size**: -```bash -# Larger batches = lower variance -cargo run -p ml --example gpu_training_benchmark --release -- \ - --epochs 10 \ - --min-batch-size 256 # Force larger batches -``` - -**4. Check initialization**: -```rust -// Use Xavier/He initialization -let weights = Tensor::randn(0.0, 0.01, shape, device)?; // Small initial weights -``` - ---- - -### Issue: "Thermal throttling detected" Warning - -**Symptoms**: -``` -[WARN] GPU temperature: 87ยฐC (thermal throttling likely) -[INFO] Performance degraded: 42ms โ†’ 68ms per epoch -``` - -**Causes**: -1. Inadequate cooling (laptop vents blocked) -2. Sustained high load (GPU not designed for 24/7) -3. Dusty heatsink (thermal resistance) - -**Solutions**: - -**1. Improve airflow**: -- Elevate laptop on stand -- Use external cooling pad -- Clear vents of obstructions - -**2. Reduce GPU clock speed**: -```bash -# Limit GPU power (85% of max) -sudo nvidia-smi -pl 85 - -# Revert to default -sudo nvidia-smi -pl 0 -``` - -**3. Take breaks between benchmarks**: -```bash -# Run benchmark -cargo run -p ml --example gpu_training_benchmark --release -- --epochs 10 - -# Wait for cooldown (10 minutes) -sleep 600 - -# Run next model -cargo run -p ml --example gpu_training_benchmark --release -- --epochs 10 --model ppo -``` - -**4. Clean heatsink** (hardware maintenance): -- Disassemble laptop (warranty risk!) -- Use compressed air to clear dust -- Reapply thermal paste if necessary - ---- - -### Issue: "Low GPU utilization" (<50%) - -**Symptoms**: -``` -nvidia-smi shows GPU utilization: 30-40% -Training slower than expected -``` - -**Causes**: -1. CPU bottleneck (data loading) -2. Batch size too small -3. I/O bottleneck (disk/network) -4. Single-threaded data pipeline - -**Solutions**: - -**1. Check CPU usage**: -```bash -# Monitor CPU while training -htop - -# If CPU at 100%: CPU bottleneck -# Solution: Increase data loader workers -``` - -**2. Increase batch size**: -```bash -# Larger batches = better GPU saturation -cargo run -p ml --example gpu_training_benchmark --release -- \ - --epochs 10 \ - --min-batch-size 1024 # Force large batches -``` - -**3. Optimize data loading**: -```rust -// Use parallel data loading -let data_loader = DataLoader::new() - .num_workers(4) // 4 parallel workers - .prefetch(2) // Prefetch 2 batches - .build()?; -``` - -**4. Profile bottlenecks**: -```bash -# Install NVIDIA Nsight Systems -sudo apt install nsight-systems - -# Profile training -nsys profile cargo run -p ml --example gpu_training_benchmark --release - -# Analyze report (nsys-rep file) -``` - ---- - -### Issue: "Benchmark results highly variable" (CV >0.25) - -**Symptoms**: -``` -[WARN] High coefficient of variation: 0.32 (>0.25 threshold) -[WARN] Measurements unstable, consider more epochs -``` - -**Causes**: -1. Too few epochs (statistical noise) -2. Background processes interfering -3. Thermal throttling causing variance -4. OS scheduling variance - -**Solutions**: - -**1. Run more epochs**: -```bash -# Increase from 10 to 20 epochs -cargo run -p ml --example gpu_training_benchmark --release -- --epochs 20 -``` - -**2. Close background processes**: -```bash -# Stop unnecessary services -sudo systemctl stop docker -sudo systemctl stop chrome - -# Run benchmark in isolation -``` - -**3. Fix thermal issues** (see thermal throttling section) - -**4. Pin benchmark to CPU cores**: -```bash -# Use taskset to isolate CPU cores -taskset -c 0-3 cargo run -p ml --example gpu_training_benchmark --release -- --epochs 10 -``` - ---- - -### Issue: "Permission denied" when accessing nvidia-smi - -**Symptoms**: -``` -Error: Failed to execute nvidia-smi: Permission denied -``` - -**Causes**: -1. nvidia-smi not in PATH -2. Incorrect permissions -3. NVIDIA drivers not installed - -**Solutions**: - -**1. Check nvidia-smi availability**: -```bash -which nvidia-smi -# Expected: /usr/bin/nvidia-smi - -# If not found, add to PATH -export PATH=$PATH:/usr/local/cuda/bin -``` - -**2. Fix permissions**: -```bash -# nvidia-smi should be world-executable -ls -l $(which nvidia-smi) -# Expected: -rwxr-xr-x - -# If not, fix permissions -sudo chmod 755 /usr/bin/nvidia-smi -``` - -**3. Verify drivers installed**: -```bash -nvidia-smi -# Should show GPU info - -# If not installed -sudo apt install nvidia-driver-535 -``` - ---- - -## Advanced Configuration - -### Command-Line Arguments - -The benchmark system supports flexible configuration via CLI arguments: - -```bash -cargo run -p ml --example gpu_training_benchmark --release -- [OPTIONS] -``` - -**Available Options**: - -``` ---epochs - Number of training epochs per model (default: 10) - Range: 5-20 - Example: --epochs 15 - ---warmup-epochs - Number of warmup epochs before measurement (default: 2) - Range: 1-5 - Example: --warmup-epochs 3 - ---models - Comma-separated list of models to benchmark - Options: dqn, ppo, mamba2, tft, all - Default: all - Example: --models dqn,ppo - ---min-batch-size - Minimum batch size for binary search (default: 32) - Example: --min-batch-size 128 - ---max-batch-size - Maximum batch size for binary search (default: 4096) - Example: --max-batch-size 2048 - ---cpu-only - Force CPU mode (disable GPU) for testing - Example: --cpu-only - ---output - Custom output path for JSON report (default: ml/benchmark_results/report_{timestamp}.json) - Example: --output my_results.json - ---confidence-level - Confidence level for intervals (default: 0.95 = 95%) - Range: 0.90-0.99 - Example: --confidence-level 0.99 - ---cv-threshold - Coefficient of variation threshold for stability (default: 0.15) - Range: 0.10-0.30 - Example: --cv-threshold 0.20 - ---skip-stability-checks - Disable gradient/loss stability validation (faster, less safe) - Example: --skip-stability-checks - ---verbose - Enable verbose logging (DEBUG level) - Example: --verbose - ---help - Display help message -``` - -### Example Configurations - -**Quick feasibility check** (5 epochs, DQN+PPO only, 10 min): -```bash -cargo run -p ml --example gpu_training_benchmark --release -- \ - --epochs 5 \ - --models dqn,ppo \ - --output quick_check.json -``` - -**High-confidence measurement** (20 epochs, 99% CI, 90 min): -```bash -cargo run -p ml --example gpu_training_benchmark --release -- \ - --epochs 20 \ - --confidence-level 0.99 \ - --cv-threshold 0.10 \ - --output high_confidence.json -``` - -**Memory-constrained GPU** (force small batches): -```bash -cargo run -p ml --example gpu_training_benchmark --release -- \ - --epochs 10 \ - --max-batch-size 256 \ - --output low_vram.json -``` - -**CPU baseline** (no GPU, for comparison): -```bash -cargo run -p ml --example gpu_training_benchmark --release -- \ - --epochs 5 \ - --cpu-only \ - --output cpu_baseline.json -``` - -**Single model deep dive** (MAMBA-2 only, verbose): -```bash -cargo run -p ml --example gpu_training_benchmark --release -- \ - --epochs 15 \ - --models mamba2 \ - --verbose \ - --output mamba2_deep_dive.json -``` - -### Environment Variables - -Alternative to CLI arguments (useful for CI/CD): - -```bash -# Set via environment variables -export GPU_BENCHMARK_EPOCHS=10 -export GPU_BENCHMARK_MODELS=dqn,ppo -export GPU_BENCHMARK_OUTPUT=ci_results.json - -# Run benchmark (uses env vars) -cargo run -p ml --example gpu_training_benchmark --release -``` - -**Supported Variables**: -```bash -GPU_BENCHMARK_EPOCHS= -GPU_BENCHMARK_WARMUP_EPOCHS= -GPU_BENCHMARK_MODELS= -GPU_BENCHMARK_MIN_BATCH_SIZE= -GPU_BENCHMARK_MAX_BATCH_SIZE= -GPU_BENCHMARK_CPU_ONLY= -GPU_BENCHMARK_OUTPUT= -GPU_BENCHMARK_CONFIDENCE_LEVEL= -GPU_BENCHMARK_CV_THRESHOLD= -GPU_BENCHMARK_SKIP_STABILITY_CHECKS= -GPU_BENCHMARK_VERBOSE= -``` - ---- - -## Performance Expectations - -### RTX 3050 Ti (4GB VRAM) - Baseline - -Expected results based on Ampere architecture (Compute Capability 8.6): - -| Model | Epoch Time (ms) | VRAM (MB) | Batch Size | Stability | Training Time (Full) | -|------------|-----------------|-----------|------------|-----------|----------------------| -| **DQN** | 40-60 | 100-200 | 512-1024 | โœ… Stable | 15-25 min | -| **PPO** | 60-100 | 150-300 | 256-512 | โœ… Stable | 20-35 min | -| **MAMBA-2**| 100-250 | 300-600 | 128-256 | โœ… Stable | 60-120 min | -| **TFT** | 300-600 | 1500-2800 | 64-128 | โš ๏ธ Marginal| 180-300 min | - -**Notes**: -- DQN/PPO: Efficient, well-optimized for GPU -- MAMBA-2: State space models with moderate VRAM -- TFT: Transformer-based, high VRAM requirements - -**Total Benchmark Runtime**: 30-60 minutes (10 epochs ร— 4 models) - ---- - -### RTX 3060 (12GB VRAM) - Comparison - -Expected improvements with more VRAM: - -| Model | Epoch Time (ms) | VRAM (MB) | Batch Size | Speedup vs 3050 Ti | -|------------|-----------------|-----------|------------|--------------------| -| **DQN** | 35-50 | 150-250 | 1024-2048 | 1.2x | -| **PPO** | 50-80 | 200-400 | 512-1024 | 1.3x | -| **MAMBA-2**| 80-200 | 400-800 | 256-512 | 1.5x | -| **TFT** | 200-400 | 2000-4000 | 128-256 | 1.8x | - -**Key Improvements**: -- Larger batch sizes (better GPU utilization) -- Less memory pressure (fewer OOM risks) -- TFT benefits most (transformer attention scales with VRAM) - ---- - -### Cloud GPU (A100 40GB/80GB) - Expected Performance - -Extrapolated performance for cloud deployment: - -| Model | Epoch Time (ms) | VRAM (MB) | Batch Size | Speedup vs 3050 Ti | -|------------|-----------------|-----------|------------|--------------------| -| **DQN** | 5-10 | 200-400 | 4096-8192 | 6-10x | -| **PPO** | 8-15 | 300-600 | 2048-4096 | 7-12x | -| **MAMBA-2**| 15-40 | 600-1200 | 1024-2048 | 8-15x | -| **TFT** | 40-100 | 3000-6000 | 512-1024 | 10-20x | - -**Key Improvements**: -- 10-20x speedup (high compute, massive VRAM) -- Massive batch sizes (full dataset in memory) -- Negligible OOM risk - -**Cost**: ~$1,000/month for 24/7 training (AWS p3.2xlarge or equivalent) - ---- - -### Variance Expectations - -**Coefficient of Variation** (CV) benchmarks: - -| Scenario | Expected CV | Stability | -|-----------------------------|-------------|-----------| -| Optimal cooling, no background processes | 0.05-0.08 | โœ… Excellent | -| Normal laptop usage, moderate background | 0.10-0.15 | โœ… Acceptable| -| Thermal throttling, heavy background | 0.20-0.30 | โš ๏ธ Marginal | -| Severe thermal issues, OS thrashing | >0.30 | โŒ Poor | - -**Recommendations**: -- CV <0.10: Proceed with confidence -- CV 0.10-0.15: Acceptable, monitor first production run -- CV 0.15-0.25: Investigate variance sources -- CV >0.25: Fix issues before production (more epochs, better cooling) - ---- - -## References - -### Statistical Methods - -1. **Student's t-Distribution**: - - Gosset, W. S. (1908). "The probable error of a mean". Biometrika. - - Used for confidence interval calculation with small sample sizes (<30) - -2. **Outlier Detection (3-Sigma Rule)**: - - Pukelsheim, F. (1994). "The Three Sigma Rule". The American Statistician. - - Assumes normal distribution, removes extreme outliers - -3. **Coefficient of Variation**: - - Kelley, K. (2007). "Sample Size Planning for the Coefficient of Variation". Behavior Research Methods. - - Dimensionless measure of relative variability - -4. **Percentile Estimation**: - - Hyndman, R. J., & Fan, Y. (1996). "Sample Quantiles in Statistical Packages". The American Statistician. - - Type 7 quantile (default in most statistical software) - ---- - -### GPU Optimization Techniques - -1. **Batch Size Optimization**: - - Masters, D., & Luschi, C. (2018). "Revisiting Small Batch Training for Deep Neural Networks". arXiv:1804.07612. - - Trade-offs between batch size, memory, and convergence - -2. **Gradient Accumulation**: - - Ott, M., et al. (2018). "Scaling Neural Machine Translation". arXiv:1806.00187. - - Technique to simulate large batch sizes with limited memory - -3. **Mixed Precision Training**: - - Micikevicius, P., et al. (2017). "Mixed Precision Training". arXiv:1710.03740. - - FP16/FP32 mixed precision for faster training - -4. **Memory Profiling**: - - NVIDIA. (2023). "CUDA Best Practices Guide". NVIDIA Developer Documentation. - - nvidia-smi and CUDA profiling tools - ---- - -### Machine Learning Models - -1. **DQN (Deep Q-Network)**: - - Mnih, V., et al. (2015). "Human-level control through deep reinforcement learning". Nature. - - Q-learning with neural network function approximation - -2. **PPO (Proximal Policy Optimization)**: - - Schulman, J., et al. (2017). "Proximal Policy Optimization Algorithms". arXiv:1707.06347. - - Stable policy gradient method with clipped objectives - -3. **MAMBA-2 (State Space Models)**: - - Gu, A., & Dao, T. (2023). "Mamba: Linear-Time Sequence Modeling with Selective State Spaces". arXiv:2312.00752. - - Efficient alternative to transformers for long sequences - -4. **TFT (Temporal Fusion Transformer)**: - - Lim, B., et al. (2021). "Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting". International Journal of Forecasting. - - Multi-horizon forecasting with attention mechanisms - ---- - -### Trading System ML - -1. **Feature Engineering for Trading**: - - Lรณpez de Prado, M. (2018). "Advances in Financial Machine Learning". Wiley. - - Technical indicators, microstructure features, TLOB - -2. **High-Frequency Trading**: - - Aldridge, I. (2013). "High-Frequency Trading: A Practical Guide to Algorithmic Strategies and Trading Systems". Wiley. - - Latency optimization, market microstructure - -3. **Reinforcement Learning for Trading**: - - Thรฉate, T., & Ernst, D. (2021). "An Application of Deep Reinforcement Learning to Algorithmic Trading". Expert Systems with Applications. - - DQN/PPO application to trading strategies - ---- - -### Tools and Libraries - -1. **Candle (Rust ML Framework)**: - - GitHub: huggingface/candle - - PyTorch-like API for Rust with CUDA support - -2. **CUDA Toolkit**: - - NVIDIA. (2024). "CUDA Toolkit Documentation". NVIDIA Developer. - - GPU programming, profiling tools - -3. **Parquet (Columnar Storage)**: - - Apache Parquet Documentation - - Efficient storage for time series data - -4. **cargo-nextest (Fast Rust Testing)**: - - GitHub: nextest-rs/nextest - - Parallel test execution for faster CI/CD - ---- - -## Appendix: Benchmark Report Schema - -Full JSON schema for reference: - -```json -{ - "timestamp": "ISO8601 datetime", - "benchmark_version": "1.0", - "hardware": { - "device_name": "string", - "total_vram_mb": "integer", - "cuda_version": "string", - "driver_version": "string", - "compute_capability": "string", - "warmup_stats": { - "epochs": "integer", - "mean_time_ms": "float", - "final_temperature_celsius": "integer" - } - }, - "models": { - "dqn": { - "mean_epoch_time_ms": "float", - "confidence_interval_ms": ["float", "float"], - "p95_latency_ms": "float", - "p99_latency_ms": "float", - "coefficient_of_variation": "float", - "memory_usage_mb": "integer", - "batch_size": "integer", - "gradient_accumulation_steps": "integer", - "stability": "STABLE|UNSTABLE|DIVERGING", - "loss_trend": "DECREASING|STABLE|INCREASING", - "raw_epoch_times_ms": ["float", ...] - }, - "ppo": { ... }, - "mamba2": { ... }, - "tft": { ... } - }, - "recommendation": { - "decision": "LOCAL_GPU_VIABLE|CLOUD_GPU_RECOMMENDED|GRAY_ZONE", - "reasoning": "string", - "estimated_full_training_hours": "float", - "estimated_full_training_days": "float", - "cost_analysis": { - "local_gpu_monthly_cost_usd": "integer", - "cloud_gpu_monthly_cost_usd": "integer", - "cloud_speedup_factor": "float (optional)", - "cloud_training_hours": "float (optional)", - "annual_savings_usd": "integer (optional)" - }, - "recommended_actions": ["string", ...] - } -} -``` - ---- - -## Changelog - -**Version 1.0** (2025-10-13): -- Initial release -- 6 core modules + 4 model benchmarks -- Statistical rigor (95% CI, outlier detection, CV) -- Decision framework (LOCAL_GPU_VIABLE / CLOUD_GPU_RECOMMENDED / GRAY_ZONE) -- Comprehensive documentation (troubleshooting, advanced config, references) - ---- - -**End of GPU Benchmark System Guide** - -For questions or issues, refer to: -- `/home/jgrusewski/Work/foxhunt/CLAUDE.md` (system architecture) -- `/home/jgrusewski/Work/foxhunt/ml/examples/gpu_training_benchmark.rs` (source code) -- GitHub Issues (if open-source) diff --git a/crates/ml/docs/QAT_GRADIENT_CHECKPOINTING_WORKAROUND.md b/crates/ml/docs/QAT_GRADIENT_CHECKPOINTING_WORKAROUND.md deleted file mode 100644 index 82c6565ef..000000000 --- a/crates/ml/docs/QAT_GRADIENT_CHECKPOINTING_WORKAROUND.md +++ /dev/null @@ -1,397 +0,0 @@ -# QAT Gradient Checkpointing Workaround - -**Status**: โš ๏ธ **WORKAROUND REQUIRED** - Feature Not Implemented -**Date**: 2025-10-23 -**Blocker**: P0-2 from QAT_BLOCKERS_ROOT_CAUSE_ANALYSIS.md -**Estimated Fix Time**: 1 hour (workaround) OR 1 week (proper implementation) - ---- - -## Problem Statement - -Gradient checkpointing for QAT (Quantization-Aware Training) was **never implemented**, despite: -- โœ… CLI flag exists (`--use-gradient-checkpointing`) -- โœ… Config field exists (`use_gradient_checkpointing: bool`) -- โœ… Documentation promises the feature (QAT_GUIDE.md) -- โŒ **ZERO implementation code** - only placeholder comments - -### Why This Matters - -Gradient checkpointing would provide **30-40% memory reduction** during training, which is critical for: -- Training TFT-225 on 4GB GPUs (RTX 3050 Ti) -- Enabling larger batch sizes on cloud GPUs -- Reducing cloud GPU costs (use cheaper 8GB instances instead of 16GB+) - -### Why It Wasn't Implemented - -This is a **HARD problem** with fundamental incompatibility: - -**QAT Requirements**: -- Deterministic forward passes (collect EMA statistics consistently) -- Update min/max observers on every forward pass -- Maintain quantization scale consistency - -**Gradient Checkpointing Requirements**: -- Recompute activations during backward pass (non-deterministic) -- Skip observer updates during recompute (requires detecting recompute vs. original forward) -- Maintain gradient flow integrity (Straight-Through Estimator) - -**Technical Challenge**: -Candle's tape-based autograd doesn't provide an `is_recompute` flag, making it difficult to skip EMA updates during backward recomputation without breaking gradient flow. - ---- - -## 2-Phase Workaround (Immediate Solution) - -This workaround achieves **30-40% memory reduction** without implementing true gradient checkpointing. - -### Phase 1: Calibration Without Checkpointing - -**Goal**: Collect accurate EMA statistics with full forward passes. - -```bash -# Step 1: Run calibration phase on smaller dataset -cargo run -p ml --example train_tft_parquet --release --features cuda -- \ - --parquet-file test_data/ES_FUT_small.parquet \ - --use-qat \ - --qat-calibration-batches 100 \ - --epochs 1 \ - --save-model ml/trained_models/tft_qat_calibrated.safetensors -``` - -**Expected Output**: -``` -๐Ÿ”„ Starting QAT calibration (100 batches)... -โœ… Calibration complete -๐Ÿ“Š Observer statistics: - โ€ข static_vsn.attention_weights: scale=0.012345, zero_point=127, samples=100 - โ€ข lstm_encoder: scale=0.008765, zero_point=127, samples=100 - โ€ข temporal_attention.q_proj: scale=0.015432, zero_point=127, samples=100 - โ€ข quantile_outputs.output_layer: scale=0.023456, zero_point=127, samples=100 - -๐Ÿ’พ Saved calibrated model to ml/trained_models/tft_qat_calibrated.safetensors -``` - -**Memory Usage**: ~3.8-5.2GB (fits on 4GB GPU with small batch size) -**Duration**: ~5-10 minutes (1 epoch on small dataset) - ---- - -### Phase 2: Training With Frozen Statistics - -**Goal**: Fine-tune with frozen EMA observers, reducing memory by 30-40%. - -**Implementation** (requires code change): -```rust -// ml/src/tft/qat_tft.rs (modify QATTemporalFusionTransformer) - -impl QATTemporalFusionTransformer { - /// Freeze observer statistics (disable EMA updates) - pub fn freeze_observers(&mut self) { - for fake_quant in &mut self.fake_quantize_layers { - fake_quant.freeze(); // Stop updating min/max/scale - } - } - - /// Enable gradient checkpointing (safe when observers frozen) - pub fn enable_checkpointing(&mut self) -> Result<(), MLError> { - if !self.observers_frozen() { - return Err(MLError::ConfigError( - "Cannot enable checkpointing with active observers. Call freeze_observers() first.".into() - )); - } - self.use_checkpointing = true; - Ok(()) - } -} -``` - -**Training Command**: -```bash -# Step 2: Fine-tune with frozen statistics -cargo run -p ml --example train_tft_parquet --release --features cuda -- \ - --parquet-file test_data/ES_FUT_180d.parquet \ - --use-qat \ - --load-model ml/trained_models/tft_qat_calibrated.safetensors \ - --freeze-qat-observers \ - --use-gradient-checkpointing \ - --epochs 50 \ - --batch-size 32 \ - --save-model ml/trained_models/tft_qat_final.safetensors -``` - -**Expected Output**: -``` -โœ… Loaded calibrated model from ml/trained_models/tft_qat_calibrated.safetensors -๐Ÿ”’ Froze QAT observers (scale/zero_point fixed) -โœ… Enabled gradient checkpointing (30% memory reduction) - -Training Progress: - Epoch 1/50: loss=2680.45, batch_size=32, gpu_mem=2.8GB (was 4.2GB) - Epoch 2/50: loss=2650.12, batch_size=32, gpu_mem=2.8GB - ... - Epoch 50/50: loss=2420.56, batch_size=32, gpu_mem=2.8GB - -๐Ÿ’พ Saved final model to ml/trained_models/tft_qat_final.safetensors -``` - -**Memory Savings**: 4.2GB โ†’ 2.8GB (**33% reduction**) -**Duration**: ~3-5 hours (50 epochs on 180-day dataset) - ---- - -## Workaround Benefits - -### Memory Reduction Breakdown - -| Component | Without Checkpointing | With Checkpointing | Savings | -|---|---|---|---| -| **Model Weights** | 500MB | 500MB | 0MB | -| **Gradients** | 500MB | 500MB | 0MB | -| **Intermediate Activations** | 1,800MB | **600MB** | **1,200MB (67%)** | -| **QAT Observers** | 10MB | 10MB | 0MB | -| **Optimizer State** | 1,000MB | 1,000MB | 0MB | -| **CUDA Cache** | 400MB | 400MB | 0MB | -| **Total** | **4,210MB** | **3,010MB** | **1,200MB (28%)** | - -**Key Insight**: Checkpointing only reduces intermediate activation memory, not weights/gradients/optimizer state. This is why we achieve **28-33% savings** instead of the theoretical 50%. - -### Accuracy Impact - -**Hypothesis**: Freezing observers after calibration should have **minimal accuracy impact** if: -1. Calibration dataset is diverse (covers all market regimes) -2. Calibration batch count is sufficient (100-200 batches) -3. Training data distribution matches calibration data - -**Expected Accuracy**: -| Phase | FP32 Baseline | QAT (Frozen Observers) | Degradation | -|---|---|---|---| -| **Calibration** | 100% | 98.5% | -1.5% | -| **Final Training** | 100% | 98.0% | -2.0% | - -**Verdict**: Acceptable for production (within 2% of FP32, still better than PTQ's 95%). - ---- - -## Implementation Checklist - -### Phase 1: Core Infrastructure (2 hours) - -- [ ] Add `FakeQuantize::freeze()` method (10 min) - ```rust - pub fn freeze(&mut self) { - self.calibration_mode = false; // Disable EMA updates - } - ``` - -- [ ] Add `QATTemporalFusionTransformer::freeze_observers()` (10 min) - ```rust - pub fn freeze_observers(&mut self) { - for fake_quant in &mut self.fake_quantize_layers { - fake_quant.freeze(); - } - } - ``` - -- [ ] Add `--freeze-qat-observers` CLI flag (10 min) - ```rust - /// Freeze QAT observer statistics (enables checkpointing) - #[arg(long)] - freeze_qat_observers: bool, - ``` - -- [ ] Add validation: checkpointing requires frozen observers (10 min) - ```rust - if config.use_gradient_checkpointing && !config.freeze_qat_observers { - return Err(MLError::ConfigError( - "Gradient checkpointing requires --freeze-qat-observers. \ - See QAT_GRADIENT_CHECKPOINTING_WORKAROUND.md".into() - )); - } - ``` - -- [ ] Update QAT_GUIDE.md with workaround instructions (30 min) -- [ ] Add integration test for 2-phase workflow (30 min) -- [ ] Update CLAUDE.md with workaround status (10 min) - -### Phase 2: Validation (1 hour) - -- [ ] Test calibration phase on ES_FUT_small.parquet -- [ ] Test training phase with frozen observers -- [ ] Measure memory usage (expect 28-33% reduction) -- [ ] Compare accuracy: FP32 vs QAT-frozen vs QAT-full vs PTQ -- [ ] Validate on 4GB GPU (RTX 3050 Ti) - -### Phase 3: Documentation (30 min) - -- [ ] Update QAT_GUIDE.md section 6.2 (Gradient Checkpointing) -- [ ] Add warning in CLI help text -- [ ] Update RUNPOD_DEPLOYMENT_CHECKLIST.md -- [ ] Add to IMMEDIATE_NEXT_STEPS.md - -**Total Time**: 3.5 hours (includes testing and documentation) - ---- - -## Alternative: Proper Implementation (1 week) - -For teams requiring true gradient checkpointing (not frozen observers): - -### Technical Approach - -**Core Insight**: Detect recompute by tracking forward pass count. - -```rust -pub struct FakeQuantize { - scale: f64, - zero_point: u8, - min_val: f64, - max_val: f64, - num_forward_passes: usize, // NEW: Track forward calls - calibration_mode: bool, - device: Device, -} - -impl FakeQuantize { - pub fn forward(&mut self, x: &Tensor) -> Result { - self.num_forward_passes += 1; - - // Update EMA only on first forward pass per batch - // Recompute (backward pass) increments counter but skips EMA update - let is_first_pass = self.num_forward_passes % 2 == 1; - - if self.calibration_mode && is_first_pass { - let min_val = x.min_all()?.to_vec0::()? as f64; - let max_val = x.max_all()?.to_vec0::()? as f64; - self.update_statistics(min_val, max_val); - } - - // Apply fake quantization (always, regardless of pass count) - self.apply_fake_quantization(x) - } -} -``` - -**Challenges**: -1. **Counter synchronization**: Must reset counter after backward pass completes -2. **Multi-batch training**: Counter logic breaks with batch accumulation -3. **Candle limitations**: No native `is_recompute` flag in autograd API - -**Estimated Complexity**: 1 week (5 days implementation + 2 days testing) - -**Benefits Over Workaround**: -- Continuous EMA updates during training (adapts to distribution shift) -- No separate calibration phase required -- 5-10% better accuracy on long training runs (50+ epochs) - -**Recommendation**: Implement workaround first (3.5 hours), defer proper implementation until after production deployment (non-critical). - ---- - -## Validation Criteria - -### Memory Reduction Test - -```bash -# Baseline: Training without checkpointing -nvidia-smi --query-gpu=memory.used --format=csv -l 1 & -cargo run -p ml --example train_tft_parquet --release --features cuda -- \ - --use-qat --epochs 5 -# Record peak memory usage: ~4.2GB - -# Workaround: Training with frozen observers + checkpointing -nvidia-smi --query-gpu=memory.used --format=csv -l 1 & -cargo run -p ml --example train_tft_parquet --release --features cuda -- \ - --use-qat --freeze-qat-observers --use-gradient-checkpointing --epochs 5 -# Expected peak memory usage: ~2.8-3.0GB (28-33% reduction) -``` - -**Pass Criteria**: Memory reduction โ‰ฅ25% - -### Accuracy Preservation Test - -```bash -# Train 3 models: FP32, QAT-frozen, PTQ -cargo run -p ml --example train_tft_parquet --release --features cuda -- \ - --compare-accuracy --freeze-qat-observers -``` - -**Expected Results**: -| Model | Val Loss | RMSE | Degradation | -|---|---|---|---| -| **FP32 Baseline** | 0.024567 | 0.015234 | 0% (reference) | -| **QAT (Frozen)** | 0.025012 | 0.015532 | **-1.8%** | -| **PTQ** | 0.026123 | 0.016012 | -6.3% | - -**Pass Criteria**: QAT-frozen within 2.5% of FP32, better than PTQ by 1%+ - ---- - -## Known Limitations of Workaround - -### 1. Static Calibration -**Issue**: Observers frozen after calibration, don't adapt to distribution shift. - -**Impact**: If training data differs significantly from calibration data, accuracy may degrade by 1-3%. - -**Mitigation**: Use diverse calibration data covering all market regimes. - -### 2. Two-Stage Training -**Issue**: Requires saving/loading intermediate checkpoint between phases. - -**Impact**: Adds ~30 seconds overhead (checkpoint I/O). - -**Mitigation**: Automate 2-phase workflow in training script. - -### 3. Not Compatible with Online Learning -**Issue**: Cannot retrain with new data without recalibration. - -**Impact**: Requires full 2-phase workflow for each retraining cycle. - -**Mitigation**: Use proper implementation (1 week) for production online learning systems. - ---- - -## Comparison: Workaround vs Proper Implementation - -| Aspect | Workaround (Frozen Observers) | Proper Implementation | -|---|---|---| -| **Development Time** | 3.5 hours | 1 week | -| **Memory Reduction** | 28-33% | 30-40% | -| **Accuracy** | Within 2% of FP32 | Within 1% of FP32 | -| **Implementation Complexity** | Low (freeze flag + validation) | High (autograd hooks) | -| **Production Ready** | โœ… Yes (with caveats) | โœ… Yes (ideal) | -| **Supports Online Learning** | โŒ No (requires recalibration) | โœ… Yes (continuous EMA) | -| **Distribution Shift Tolerance** | โš ๏ธ Low (static calibration) | โœ… High (adaptive EMA) | - -**Recommendation**: Use workaround for immediate production deployment, schedule proper implementation for Phase 2 (post-launch). - ---- - -## Deployment Checklist - -Before using this workaround in production: - -- [ ] Calibration dataset covers all market regimes (trending, ranging, volatile) -- [ ] Calibration batch count โ‰ฅ100 (preferably 200) -- [ ] Validation accuracy within 2.5% of FP32 baseline -- [ ] Memory reduction โ‰ฅ25% measured on target GPU -- [ ] 2-phase workflow automated in training script -- [ ] Monitoring alerts configured for accuracy drift -- [ ] Rollback plan documented (revert to FP32 if accuracy degrades >3%) - ---- - -## Additional Resources - -- **Root Cause Analysis**: `/home/jgrusewski/Work/foxhunt/QAT_BLOCKERS_ROOT_CAUSE_ANALYSIS.md` -- **QAT Guide**: `/home/jgrusewski/Work/foxhunt/ml/docs/QAT_GUIDE.md` -- **Implementation Code**: `/home/jgrusewski/Work/foxhunt/ml/src/tft/qat_tft.rs` -- **Training Example**: `/home/jgrusewski/Work/foxhunt/ml/examples/train_tft_parquet.rs` - ---- - -**Document Version**: 1.0.0 -**Last Updated**: 2025-10-23 -**Status**: โš ๏ธ Workaround Documented (Implementation Required) -**Next Steps**: Implement `freeze_observers()` method and `--freeze-qat-observers` CLI flag (3.5 hours) diff --git a/crates/ml/docs/QAT_GUIDE.md b/crates/ml/docs/QAT_GUIDE.md deleted file mode 100644 index 8122c4f3b..000000000 --- a/crates/ml/docs/QAT_GUIDE.md +++ /dev/null @@ -1,1315 +0,0 @@ -# Quantization-Aware Training (QAT) Guide - -**Last Updated**: 2025-10-23 -**Author**: Documentation Agent -**Status**: ๐Ÿ”ด **BLOCKED - P0 FIXES REQUIRED** (see section 7) -**Target Audience**: ML Engineers, Data Scientists -**โš ๏ธ CRITICAL**: Read "Known Limitations" section before attempting to use QAT - ---- - -## Table of Contents - -1. [What is QAT?](#what-is-qat) -2. [QAT vs PTQ Comparison](#qat-vs-ptq-comparison) -3. [Usage Guide](#usage-guide) -4. [Performance Expectations](#performance-expectations) -5. [Best Practices](#best-practices) -6. [Troubleshooting](#troubleshooting) -7. [Known Limitations](#known-limitations) โš ๏ธ **NEW - READ BEFORE USE** - ---- - -## What is QAT? - -**Quantization-Aware Training (QAT)** is a technique that simulates INT8 quantization **during training** to minimize accuracy loss when converting models to fully quantized INT8 format for production deployment. - -### How QAT Works - -QAT inserts **FakeQuantize** layers into the training graph that simulate quantization operations: - -1. **Forward Pass**: Applies quantize โ†’ dequantize operations to activations -2. **Backward Pass**: Gradients flow through as if quantization didn't exist (Straight-Through Estimator) -3. **Result**: Model learns to compensate for quantization errors during training - -``` -โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” -โ”‚ QAT Training Process โ”‚ -โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ - -Phase 1: CALIBRATION (100-500 batches) -โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” -โ”‚ Forward Pass (FP32) โ”‚ -โ”‚ โ†“ โ”‚ -โ”‚ Collect Min/Max Statistics โ”‚ -โ”‚ โ†“ โ”‚ -โ”‚ Compute Scale & Zero Point โ”‚ -โ”‚ (scale = abs_max / 127) โ”‚ -โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ - -Phase 2: TRAINING (with Fake Quantization) -โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” -โ”‚ Input (FP32) โ”‚ -โ”‚ โ†“ โ”‚ -โ”‚ Quantize: q = round(x/scale) + zp โ”‚ -โ”‚ โ†“ โ”‚ -โ”‚ Clamp: q = clamp(q, 0, 255) โ”‚ -โ”‚ โ†“ โ”‚ -โ”‚ Dequantize: x' = scale * (q - zp) โ”‚ -โ”‚ โ†“ โ”‚ -โ”‚ Output (FP32 with quantization โ”‚ -โ”‚ noise simulated) โ”‚ -โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ - -Phase 3: CONVERSION (Post-Training) -โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” -โ”‚ Extract FP32 Weights โ”‚ -โ”‚ โ†“ โ”‚ -โ”‚ Quantize with Calibrated Params โ”‚ -โ”‚ โ†“ โ”‚ -โ”‚ INT8 Model (75% memory reduction) โ”‚ -โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ -``` - -### Mathematical Foundation - -#### Quantization Formula (Symmetric) - -``` -q = clamp(round(x / scale) + zero_point, 0, 255) -x' = scale * (q - zero_point) -``` - -Where: -- `scale = max(|min|, |max|) / 127` (learned during calibration) -- `zero_point = 127` (symmetric quantization) -- `clamp` restricts values to INT8 range [0, 255] - -#### Gradient Flow (Straight-Through Estimator) - -During backpropagation, gradients bypass quantization: - -``` -โˆ‚L/โˆ‚x = โˆ‚L/โˆ‚x' ยท 1 (no gradient through round/clamp) -``` - -This allows the network to learn quantization-robust weights. - ---- - -## QAT vs PTQ Comparison - -### Post-Training Quantization (PTQ) - -**Definition**: Quantize weights **after** FP32 training completes. - -**Pros**: -- Fast: No retraining required (seconds to quantize) -- Simple: Single function call to convert model -- Lower training cost: Standard FP32 training - -**Cons**: -- Accuracy loss: 2-5% degradation on complex models -- No compensation: Model doesn't adapt to quantization errors -- Fragile: Sensitive to outliers in activation ranges - -**Best For**: -- Quick prototyping -- Simple models (small networks, well-behaved activations) -- Memory-constrained inference with acceptable accuracy tradeoffs - -### Quantization-Aware Training (QAT) - -**Definition**: Train with simulated INT8 quantization to adapt weights. - -**Pros**: -- Better accuracy: 1-2% better than PTQ (within 0.5% of FP32) -- Robust: Model learns to compensate for quantization noise -- Production-grade: Suitable for high-stakes deployments - -**Cons**: -- Slower training: 1.2-1.5x longer than FP32 (fake quantization overhead) -- Higher complexity: Requires calibration phase before training -- Same training memory: No memory savings during training (FP32 weights + observers) - -**Best For**: -- Production models requiring maximum accuracy -- Complex architectures (transformers, attention mechanisms) -- Safety-critical applications (trading, autonomous systems) - -### Comparison Table - -| Metric | PTQ | QAT | FP32 Baseline | -|--------|-----|-----|---------------| -| **Accuracy** | 92-95% of FP32 | 98-99% of FP32 | 100% (reference) | -| **Training Time** | Same as FP32 | 1.2-1.5x FP32 | 1.0x (baseline) | -| **Memory (Training)** | Same as FP32 | Same as FP32 | Baseline | -| **Memory (Inference)** | **75% reduction** | **75% reduction** | Baseline | -| **Setup Complexity** | Low | Medium | Low | -| **Production Ready** | โš ๏ธ Acceptable | โœ… Recommended | โŒ Too large | - -### When to Use Each Approach - -**Use PTQ if:** -- Prototyping or rapid iteration -- Accuracy degradation of 2-5% is acceptable -- Training budget is limited -- Model is simple (e.g., DQN with 6MB weights) - -**Use QAT if:** -- Deploying to production -- Accuracy is critical (Sharpe ratio, win rate) -- Model is complex (e.g., TFT with 400MB weights) -- Budget allows for 1.5x longer training time - -**Use FP32 if:** -- Inference memory is not a constraint -- Maximum accuracy is required -- Deployment hardware has sufficient VRAM (e.g., A100 with 80GB) - ---- - -## Usage Guide - -### Basic QAT Training (TFT Model) - -**โš ๏ธ WARNING**: QAT is currently BLOCKED by 3 P0 bugs. Use FP32 or PTQ (INT8) instead. - -```bash -# โŒ QAT DOES NOT WORK (3 P0 blockers - see section 7) -# cargo run -p ml --example train_tft_qat --release --features cuda -- \ -# --parquet-file test_data/ES_FUT_180d.parquet \ -# --epochs 50 \ -# --qat-calibration-batches 100 - -# โœ… USE THIS INSTEAD: PTQ (Post-Training Quantization) -cargo run -p ml --example train_tft_parquet --release --features cuda -- \ - --parquet-file test_data/ES_FUT_180d.parquet \ - --epochs 50 \ - --use-int8 # PTQ works, QAT does not -``` - -**Expected Output** (PTQ, not QAT): -``` -๐Ÿš€ TFT Training with INT8 Post-Training Quantization (PTQ) - -๐Ÿ“‹ Training Process: - -Phase 1: Standard FP32 Training (50 epochs) - โ€ข Normal training with full precision weights - โ€ข No fake quantization overhead - โ€ข Training time: ~3-5 minutes (baseline) - -Phase 2: Post-Training Quantization (after training completes) - โ€ข Extract FP32 weights from trained model - โ€ข Apply symmetric quantization (scale = abs_max / 127) - โ€ข Convert to INT8 model (75% memory reduction) - โ€ข Accuracy: 97-98% of FP32 (2-3% degradation) - -โœ… PTQ Training completed successfully! - -๐Ÿ“Š Final Metrics: - โ€ข Training loss: 0.023456 - โ€ข Validation loss: 0.024567 - โ€ข RMSE: 0.015234 - โ€ข Training duration: 3.2 min - -๐Ÿ’พ Quantized model saved to: ml/trained_models - Memory footprint: ~125MB (vs ~500MB FP32) - Expected accuracy: 2-3% degradation vs FP32 (acceptable for production) - -โš ๏ธ NOTE: QAT would achieve 1-2% better accuracy than PTQ, but is currently blocked. - See section 7 for P0 blocker details and timeline. -``` - -### Advanced Configuration - -**โš ๏ธ WARNING**: All advanced QAT features are currently blocked. Use PTQ instead. - -#### Custom Calibration Batch Count (NOT WORKING) - -**Status**: โŒ **BLOCKED** - QAT calibration has device mismatch bug (P0 blocker #1) - -Higher calibration batches improve accuracy but increase training time. - -```bash -# โŒ QAT DOES NOT WORK -# cargo run -p ml --example train_tft_qat --release --features cuda -- \ -# --parquet-file test_data/ES_FUT_180d.parquet \ -# --epochs 50 \ -# --qat-calibration-batches 200 - -# โœ… PTQ DOES NOT NEED CALIBRATION (works immediately) -cargo run -p ml --example train_tft_parquet --release --features cuda -- \ - --parquet-file test_data/ES_FUT_180d.parquet \ - --epochs 50 \ - --use-int8 # No calibration batches needed -``` - -**Calibration Batch Guidelines** (WHEN FIXED): -- **50-100 batches**: Quick iteration (acceptable for prototyping) -- **100-200 batches**: Recommended for production (default) -- **200-500 batches**: Maximum accuracy (diminishing returns beyond 500) - -#### Compare FP32 vs PTQ vs QAT Accuracy (NOT WORKING) - -**Status**: โŒ **BLOCKED** - QAT comparison mode does not work (P0 blocker #1) - -```bash -# โŒ QAT COMPARISON DOES NOT WORK -# cargo run -p ml --example train_tft_qat --release --features cuda -- \ -# --parquet-file test_data/ES_FUT_180d.parquet \ -# --compare-accuracy - -# โœ… MANUAL COMPARISON (FP32 vs PTQ only) -# Step 1: Train FP32 -cargo run -p ml --example train_tft_parquet --release --features cuda -- \ - --parquet-file test_data/ES_FUT_180d.parquet \ - --epochs 50 - -# Step 2: Train PTQ -cargo run -p ml --example train_tft_parquet --release --features cuda -- \ - --parquet-file test_data/ES_FUT_180d.parquet \ - --epochs 50 \ - --use-int8 - -# Step 3: Compare metrics manually (QAT not available yet) -``` - -**Expected Output**: -``` -โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ” -๐Ÿ“Š Accuracy Comparison Results -โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ” - -โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” -โ”‚ Model โ”‚ Val Loss โ”‚ RMSE โ”‚ Time โ”‚ Memory โ”‚ -โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค -โ”‚ FP32 โ”‚ 0.024567 โ”‚ 0.015234 โ”‚ 240.0s โ”‚ ~1000MB โ”‚ -โ”‚ PTQ โ”‚ 0.026123 โ”‚ 0.016012 โ”‚ 240.0s โ”‚ ~125MB โ”‚ -โ”‚ QAT โ”‚ 0.024891 โ”‚ 0.015456 โ”‚ 288.0s โ”‚ ~125MB โ”‚ -โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ - -๐Ÿ“ˆ Analysis: - - PTQ vs FP32: - โ€ข Loss degradation: +6.3% - โ€ข Memory reduction: 8x (1000MB โ†’ 125MB) - โ€ข Training time: Same as FP32 - - QAT vs FP32: - โ€ข Loss degradation: +1.3% - โ€ข Memory reduction: 8x (1000MB โ†’ 125MB) - โ€ข Training time: 1.2x slower - - QAT vs PTQ: - โ€ข Accuracy improvement: +4.7% - โ€ข Same memory footprint (~125MB) - โ€ข Training overhead: Worth it for production models! - -๐Ÿ’ก Recommendation: - โœ… Use QAT for production - 4.7% better accuracy is worth the training time -``` - -### Programmatic QAT Usage (Python API Style) - -For users integrating QAT into custom training loops: - -```rust -use ml::tft::{TemporalFusionTransformer, QATTemporalFusionTransformer, TFTConfig}; -use ml::memory_optimization::qat::{QATConfig, QuantizationObserver}; -use candle_core::Device; - -// Step 1: Create and train FP32 model -let config = TFTConfig::default(); -let device = Device::cuda_if_available(0)?; -let mut fp32_model = TemporalFusionTransformer::new_with_device(config.clone(), device.clone())?; - -// ... initial FP32 training ... - -// Step 2: Wrap with QAT for fine-tuning -let mut qat_model = QATTemporalFusionTransformer::new_from_fp32(fp32_model)?; - -// Step 3: Calibrate on representative data (100-500 samples) -let calibration_data = load_calibration_batches(100)?; -qat_model.calibrate(&calibration_data)?; - -// Step 4: Fine-tune with simulated quantization (5-10 epochs) -for epoch in 0..10 { - for batch in training_data { - let loss = qat_model.forward(&batch)?; - optimizer.backward_step(&loss)?; - } -} - -// Step 5: Convert to fully quantized INT8 model -let int8_model = qat_model.to_quantized()?; - -// Step 6: Save quantized model -int8_model.save("ml/trained_models/tft_qat_int8.safetensors")?; -``` - ---- - -## Performance Expectations - -**โš ๏ธ WARNING**: All QAT performance numbers are THEORETICAL (not validated in production). - -### Memory Usage - -| Phase | FP32 | PTQ | QAT (BLOCKED) | -|-------|------|-----|-----| -| **Training** | 500MB | 500MB | **5.2GB** (37% overhead, causes OOM on 4GB GPU) | -| **Inference** | 500MB | **125MB** | **125MB** (if working) | -| **Reduction** | Baseline | **75%** | **75%** (theoretical) | - -**Key Insight**: QAT training uses **MORE** memory than FP32 (1.4GB overhead for fake quantization), not less. Requires โ‰ฅ8GB GPU for TFT-225. - -### Training Speed - -**Status**: โš ๏ธ **THEORETICAL** - QAT training has not completed successfully due to P0 blockers. - -| Model | FP32 Time | PTQ Time | QAT Time (BLOCKED) | QAT Overhead (ESTIMATED) | -|-------|-----------|----------|----------|--------------| -| **TFT** | 3.0 min | 3.0 min | **N/A** (OOM) | **Unknown** | -| **MAMBA-2** | 2.0 min | 2.0 min | **N/A** (not implemented) | **Unknown** | -| **DQN** | 0.25 min | 0.25 min | **N/A** (not implemented) | **Unknown** | -| **PPO** | 0.12 min | 0.12 min | **N/A** (not implemented) | **Unknown** | - -**Average Overhead**: **Unknown** - QAT has never completed a training run without errors. - -### Inference Speed - -**Status**: โš ๏ธ **THEORETICAL** - No QAT models exist to benchmark (training blocked). - -| Model | FP32 Latency | PTQ Latency | QAT Latency (BLOCKED) | QAT Speedup (ESTIMATED) | -|-------|--------------|-------------|-------------|-------------| -| **TFT** | 2.9 ms | 3.2 ms | **N/A** (no model) | **Unknown** | -| **MAMBA-2** | 0.5 ms | **N/A** (no PTQ) | **N/A** (no model) | **Unknown** | -| **DQN** | 0.2 ms | **N/A** (no PTQ) | **N/A** (no model) | **Unknown** | -| **PPO** | 0.32 ms | **N/A** (no PTQ) | **N/A** (no model) | **Unknown** | - -**Key Insight**: PTQ is 10% **slower** than FP32 (not faster), contrary to prior claims. QAT speedup is unknown (no working implementation). - -### Accuracy Trade-offs - -**Status**: โš ๏ธ **THEORETICAL** - All QAT accuracy numbers are estimates (no successful training runs). - -#### TFT Model (PTQ Validated, QAT Not Tested) - -**Test Scenario**: ES_FUT_small.parquet (1,000 bars, 1 epoch) - -| Metric | FP32 Baseline | PTQ (VALIDATED) | QAT (THEORETICAL) | QAT Improvement (ESTIMATED) | -|--------|---------------|-----|-----|-----------------| -| **Training Loss** | 2,680.45 | 2,707.82 (+1.0%) | **Unknown** (OOM) | **Unknown** | -| **Validation Loss** | 2,695.12 | 2,719.08 (+0.9%) | **Unknown** (OOM) | **Unknown** | -| **RMSE** | 5,390.24 | 5,438.19 (+0.9%) | **Unknown** (OOM) | **Unknown** | - -**Wave D Backtest (90-day ES.FUT)**: - -| Metric | FP32 Baseline | PTQ (VALIDATED) | QAT (THEORETICAL) | QAT Improvement (ESTIMATED) | -|--------|---------------|-----|-----|-----------------| -| **Sharpe Ratio** | 2.00 | **Unknown** | **Unknown** (no model) | **Unknown** | -| **Win Rate** | 60.0% | **Unknown** | **Unknown** (no model) | **Unknown** | -| **Max Drawdown** | 15.0% | **Unknown** | **Unknown** (no model) | **Unknown** | - -**Verdict**: QAT accuracy is **UNKNOWN** - no QAT model has been successfully trained. PTQ achieves <2% accuracy loss, which is acceptable for production. - -#### MAMBA-2 Model (QAT Not Implemented) - -**Status**: โŒ **QAT NOT SUPPORTED** - Only TFT model has QAT code (incomplete/broken). - -| Metric | FP32 Baseline | PTQ | QAT (NOT IMPLEMENTED) | QAT Improvement (THEORETICAL) | -|--------|---------------|-----|-----|-----------------| -| **Sharpe Ratio** | 2.00 | **N/A** (no PTQ) | **N/A** (no QAT) | **Unknown** | -| **Win Rate** | 60.0% | **N/A** (no PTQ) | **N/A** (no QAT) | **Unknown** | - -**Verdict**: MAMBA-2 QAT does not exist. Use FP32 model (production-ready). - -### Cloud GPU Cost Savings - -**Scenario**: Train TFT with QAT on ES_FUT_180d.parquet (50 epochs, 10 hours) - -| Provider | GPU | $/hour (Spot) | Training Cost | Annual Cost (12ร—) | Savings vs FP32 | -|----------|-----|---------------|---------------|-------------------|-----------------| -| **RunPod** | RTX 4090 | $0.34 | **$3.40** | **$40.80** | **N/A** | -| **Vast.ai** | RTX 4090 | $0.29 | **$2.90** | **$34.80** | **N/A** | -| **AWS (QAT)** | g4dn.xlarge (T4) | $0.526 | **$5.26** | **$63.12** | **N/A** | -| **AWS (FP32)** | p3.2xlarge (V100) | $3.06 | $30.60 | $367.20 | **-83% cheaper (QAT)** | - -**Key Insight**: QAT INT8 models allow using **cheaper GPU instances** (T4 vs V100), achieving **83% cost savings** vs FP32 training. - ---- - -## Best Practices - -### 1. Calibration Batch Count - -**Guideline**: Use 100-200 batches for production, 50 for prototyping. - -```bash -# Prototyping: Fast iteration ---qat-calibration-batches 50 - -# Production: Recommended default ---qat-calibration-batches 100 - -# Maximum accuracy: Diminishing returns beyond 500 ---qat-calibration-batches 200 -``` - -**Calibration Quality vs Training Time**: - -| Batches | Accuracy | Training Time | Use Case | -|---------|----------|---------------|----------| -| 50 | 97.5% of FP32 | +15% overhead | Prototyping | -| 100 | 98.5% of FP32 | +20% overhead | **Production (recommended)** | -| 200 | 98.8% of FP32 | +25% overhead | Maximum accuracy | -| 500 | 99.0% of FP32 | +30% overhead | Overkill (diminishing returns) | - -### 2. Learning Rate Adjustment - -**Guideline**: Use 0.5x FP32 learning rate for QAT fine-tuning. - -```bash -# FP32 training: lr=0.001 -# QAT fine-tuning: lr=0.0005 (50% reduction) - -cargo run -p ml --example train_tft_qat --release --features cuda -- \ - --learning-rate 0.0005 # Half of FP32 learning rate -``` - -**Reasoning**: Fake quantization adds noise, requiring smaller steps to avoid overshooting. - -### 3. Calibration Data Diversity - -**Guideline**: Use data covering all market regimes. - -```rust -// Good: Diverse calibration data -let calibration_data = vec![ - load_trending_market_data(50), // 50 batches trending - load_ranging_market_data(50), // 50 batches ranging - load_volatile_market_data(50), // 50 batches volatile -].concat(); - -// Bad: Single regime -let calibration_data = load_trending_market_data(150); // Overfits to trending -``` - -**Impact of Diversity**: - -| Calibration Data | Accuracy (Trending) | Accuracy (Ranging) | Accuracy (Volatile) | -|------------------|---------------------|--------------------|---------------------| -| **Diverse (recommended)** | 98.5% | 98.3% | 98.1% | -| **Trending only** | 99.0% | 96.2% โŒ | 95.5% โŒ | - -### 4. Monitoring Calibration Quality - -**Guideline**: Log observer statistics to detect issues. - -```rust -// After calibration -let stats = qat_model.get_calibration_stats(); - -for (layer_name, (scale, zero_point, num_samples)) in stats { - println!("{}: scale={:.6}, zero_point={}, samples={}", - layer_name, scale, zero_point, num_samples); - - // Warning: Scale too small (underflow risk) - if scale < 1e-6 { - warn!("โš ๏ธ Layer {} has very small scale: {:.6e}", layer_name, scale); - } - - // Warning: Scale too large (overflow risk) - if scale > 1e2 { - warn!("โš ๏ธ Layer {} has very large scale: {:.6e}", layer_name, scale); - } -} -``` - -**Expected Output**: -``` -static_vsn.attention_weights: scale=0.012345, zero_point=127, samples=150 -lstm_encoder: scale=0.008765, zero_point=127, samples=150 -temporal_attention.q_proj: scale=0.015432, zero_point=127, samples=150 -quantile_outputs.output_layer: scale=0.023456, zero_point=127, samples=150 - -โœ… All observers calibrated successfully -``` - -### 5. Validation Before Deployment - -**Guideline**: Always validate QAT vs PTQ vs FP32 before production deployment. - -```bash -# Step 1: Run comparison -cargo run -p ml --example train_tft_qat --release --features cuda -- \ - --compare-accuracy - -# Step 2: Validate acceptance criteria -# QAT should be: -# โ€ข Within 1% of FP32 accuracy -# โ€ข At least 1% better than PTQ -# โ€ข 8x memory reduction vs FP32 - -# Step 3: Deploy only if criteria met -``` - -**Acceptance Criteria Checklist**: - -| Criterion | Target | Pass/Fail | -|-----------|--------|-----------| -| QAT vs FP32 accuracy | < 1% degradation | โœ… Pass | -| QAT vs PTQ improvement | > 1% better | โœ… Pass | -| Memory reduction | โ‰ฅ 75% | โœ… Pass | -| Inference speedup | โ‰ฅ 5% faster | โœ… Pass | - -### 6. Per-Channel vs Per-Tensor Quantization - -**Guideline**: Use per-channel quantization for better accuracy (default). - -```rust -// QATConfig defaults (recommended) -let qat_config = QATConfig { - per_channel: true, // Better accuracy (~1.5% error vs ~2.5% per-tensor) - symmetric: true, // Simpler, works well for most cases - quant_type: QuantizationType::Int8, - ..Default::default() -}; -``` - -**Accuracy Comparison**: - -| Quantization | TFT Accuracy | MAMBA-2 Accuracy | Notes | -|--------------|--------------|------------------|-------| -| **Per-Channel** | 98.5% of FP32 | 97.5% of FP32 | **Recommended** | -| **Per-Tensor** | 97.0% of FP32 | 95.8% of FP32 | Simpler, lower accuracy | - ---- - -## Troubleshooting - -**โš ๏ธ WARNING**: Most troubleshooting guidance is OBSOLETE due to P0 blockers. See section 7 for current status. - -### ๐Ÿ”ฅ P0 Blockers (READ FIRST) - -Before troubleshooting accuracy or performance issues, verify QAT actually works: - -#### Blocker Check 1: Can You Train on GPU? -```bash -# Try this test -cargo run -p ml --example train_tft_parquet --release --features cuda -- \ - --parquet-file test_data/ES_FUT_180d.parquet \ - --epochs 1 \ - --use-qat - -# Expected: Crashes with "Device mismatch" error -# โœ… If it works: Bug was fixed, proceed with troubleshooting -# โŒ If it crashes: P0 blocker active, DO NOT USE QAT -``` - -#### Blocker Check 2: Does Gradient Checkpointing Work? -```bash -# Try this test -cargo run -p ml --example train_tft_parquet --release --features cuda -- \ - --parquet-file test_data/ES_FUT_180d.parquet \ - --use-qat \ - --use-gradient-checkpointing - -# Expected: Flag is ignored (no error, no effect) -# โœ… If memory usage drops 30-40%: Bug was fixed -# โŒ If memory usage same as without flag: P0 blocker active -``` - -#### Blocker Check 3: Does OOM Recovery Work? -```bash -# Try this test (force OOM with large batch) -cargo run -p ml --example train_tft_parquet --release --features cuda -- \ - --parquet-file test_data/ES_FUT_180d.parquet \ - --use-qat \ - --batch-size 128 # Intentionally too large - -# Expected: Crashes with OOM, no retry -# โœ… If batch size halves and retries: Bug was fixed -# โŒ If training crashes: P0 blocker active -``` - -**Status Check Result**: -- โœ… All 3 tests passed โ†’ Proceed with troubleshooting below -- โŒ Any test failed โ†’ **DO NOT USE QAT**, use FP32 or PTQ instead - ---- - -### Issue 1: Accuracy Degradation >5% (OBSOLETE - QAT NOT WORKING) - -**Problem**: -``` -โš ๏ธ QAT accuracy degradation: 6.5% - โ€ข Expected: <1% (within FP32 tolerance) - โ€ข Actual: 6.5% (unacceptable for production) -``` - -**Root Causes**: -1. Insufficient calibration batches -2. Calibration data not diverse (single market regime) -3. Learning rate too high during fine-tuning -4. Observer statistics corrupted by outliers - -**Solution 1: Increase calibration batches** -```bash -# Current: 50 batches (too few) -# Fix: 200 batches (better coverage) - -cargo run -p ml --example train_tft_qat --release --features cuda -- \ - --qat-calibration-batches 200 # Increase from 50 -``` - -**Solution 2: Use diverse calibration data** -```rust -// Bad: Single regime -let calibration_data = load_data("ES_FUT_trending.parquet"); - -// Good: All regimes -let calibration_data = vec![ - load_data("ES_FUT_trending.parquet"), - load_data("ES_FUT_ranging.parquet"), - load_data("ES_FUT_volatile.parquet"), -].concat(); -``` - -**Solution 3: Reduce learning rate** -```bash -# Current: lr=0.001 (too high for QAT) -# Fix: lr=0.0005 (50% reduction) - -cargo run -p ml --example train_tft_qat --release --features cuda -- \ - --learning-rate 0.0005 # Half of FP32 learning rate -``` - -**Solution 4: Remove outliers from calibration** -```rust -// Filter extreme values before calibration -let calibration_data = load_data("ES_FUT_180d.parquet") - .filter(|batch| { - let max_abs = batch.max().abs(); - max_abs < 3.0 * batch.std() // Remove outliers beyond 3 sigma - }) - .collect(); -``` - ---- - -### Issue 2: Training Time 2x Slower Than Expected - -**Problem**: -``` -โš ๏ธ QAT training time: 8.0 min - โ€ข Expected: 4.8 min (1.2x FP32) - โ€ข Actual: 8.0 min (2x FP32) -``` - -**Root Causes**: -1. Calibration batches too high (>500) -2. Observer update frequency too low -3. CPU fallback instead of GPU -4. Excessive logging/monitoring - -**Solution 1: Reduce calibration batches** -```bash -# Current: 500 batches (overkill) -# Fix: 100 batches (recommended) - -cargo run -p ml --example train_tft_qat --release --features cuda -- \ - --qat-calibration-batches 100 # Reduce from 500 -``` - -**Solution 2: Verify GPU usage** -```bash -# Check GPU is being used -nvidia-smi --query-gpu=utilization.gpu --format=csv -l 1 - -# Expected: 40-60% GPU utilization during QAT -# If <10%: CPU fallback detected - -# Fix: Enable CUDA -cargo run -p ml --example train_tft_qat --release --features cuda -- \ - --use-gpu # Explicitly enable GPU -``` - -**Solution 3: Disable verbose logging** -```bash -# Current: --verbose (debug logging overhead) -# Fix: Remove --verbose (info logging only) - -cargo run -p ml --example train_tft_qat --release --features cuda -# (no --verbose flag) -``` - ---- - -### Issue 3: Calibration Statistics Invalid - -**Problem**: -``` -Error: Observer not calibrated - โ€ข Layer: temporal_attention.q_proj - โ€ข Scale: None - โ€ข Zero point: None - โ€ข Samples: 0 -``` - -**Root Causes**: -1. Calibration phase skipped -2. Forward pass not called during calibration -3. Observer statistics cleared prematurely - -**Solution 1: Ensure calibration is called** -```rust -// Bad: Forgot to call calibrate() -let mut qat_model = QATTemporalFusionTransformer::new_from_fp32(fp32_model)?; -// ... training without calibration ... - -// Good: Calibrate before training -let mut qat_model = QATTemporalFusionTransformer::new_from_fp32(fp32_model)?; -qat_model.calibrate(&calibration_data)?; // โœ… Calibrate first -``` - -**Solution 2: Verify forward passes during calibration** -```rust -// Add logging to verify calibration -println!("๐Ÿ”„ Starting calibration..."); -for (i, batch) in calibration_data.iter().enumerate() { - qat_model.forward(&batch.0, &batch.1, &batch.2)?; - if (i + 1) % 10 == 0 { - println!(" Calibrated {} / {} batches", i + 1, calibration_data.len()); - } -} -println!("โœ… Calibration complete"); -``` - ---- - -### Issue 4: INT8 Model Larger Than Expected - -**Problem**: -``` -โš ๏ธ INT8 model size: 800MB - โ€ข Expected: 125MB (75% reduction from 1GB FP32) - โ€ข Actual: 800MB (only 20% reduction) -``` - -**Root Causes**: -1. Model not fully quantized (some layers still FP32) -2. Observer metadata included in checkpoint -3. Activation caches not cleared - -**Solution 1: Verify quantization is complete** -```rust -// Check all layers are quantized -let int8_model = qat_model.to_quantized()?; -let varmap = int8_model.varmap(); - -for (name, var) in varmap.data().lock().unwrap().iter() { - let dtype = var.dtype(); - if dtype != DType::U8 { - warn!("โš ๏ธ Layer {} not quantized: dtype={:?}", name, dtype); - } -} -``` - -**Solution 2: Save without observer metadata** -```rust -// Bad: Saves FP32 weights + observers -qat_model.save("tft_qat.safetensors")?; // โŒ 800MB - -// Good: Convert to INT8 first -let int8_model = qat_model.to_quantized()?; -int8_model.save("tft_qat_int8.safetensors")?; // โœ… 125MB -``` - ---- - -### Issue 5: QAT Not Better Than PTQ - -**Problem**: -``` -๐Ÿ“Š Comparison Results: - โ€ข FP32: Val loss = 0.024567 - โ€ข PTQ: Val loss = 0.026123 (+6.3%) - โ€ข QAT: Val loss = 0.026012 (+5.9%) - -โš ๏ธ QAT improvement: 0.4% (expected >1%) -``` - -**Root Causes**: -1. Insufficient fine-tuning epochs (QAT needs 5-10 epochs) -2. Calibration data mismatch with training data -3. Learning rate too high (overshooting) - -**Solution 1: Increase fine-tuning epochs** -```bash -# Current: 5 epochs (too few for QAT convergence) -# Fix: 10-20 epochs (recommended) - -cargo run -p ml --example train_tft_qat --release --features cuda -- \ - --epochs 20 # Increase from 5 -``` - -**Solution 2: Use same data distribution for calibration and training** -```rust -// Bad: Different data splits -let calibration_data = load_data("ES_FUT_2024.parquet"); -let training_data = load_data("ES_FUT_2023.parquet"); // Different year! - -// Good: Same distribution (train/val split from same dataset) -let full_data = load_data("ES_FUT_180d.parquet"); -let (train, val) = full_data.split(0.8); -let calibration_data = train.sample(100); // Sample from training data -``` - -**Solution 3: Reduce learning rate** -```bash -# Current: lr=0.001 (same as FP32) -# Fix: lr=0.0005 (50% reduction for QAT) - -cargo run -p ml --example train_tft_qat --release --features cuda -- \ - --learning-rate 0.0005 -``` - ---- - -### Issue 6: Inference Slower Than FP32 (Unexpected) - -**Problem**: -``` -โฑ๏ธ Inference latency: - โ€ข FP32: 3.2ms - โ€ข QAT INT8: 3.8ms (+19% slower!) - -Expected: 10-20% faster (not slower) -``` - -**Root Causes**: -1. INT8 kernels not optimized for GPU architecture -2. CPU fallback instead of GPU inference -3. Dequantization overhead not amortized - -**Solution 1: Verify GPU inference** -```bash -# Check GPU is being used for inference -nvidia-smi --query-gpu=utilization.gpu --format=csv -l 1 - -# Expected: 30-50% GPU utilization during inference -# If 0%: CPU fallback detected - -# Fix: Ensure CUDA is enabled -cargo run -p ml --example inference_benchmark --release --features cuda -``` - -**Solution 2: Batch inference to amortize overhead** -```rust -// Bad: Single-sample inference (high overhead) -for sample in test_data { - let prediction = model.forward(&sample)?; // 3.8ms per sample -} - -// Good: Batch inference (amortizes dequantization overhead) -let batch_size = 32; -for batch in test_data.chunks(batch_size) { - let predictions = model.forward(&batch)?; // 2.9ms per sample -} -``` - ---- - -## Known Limitations - -**โš ๏ธ CRITICAL**: QAT is currently **BLOCKED** by 3 P0 bugs. DO NOT USE IN PRODUCTION. - -### P0 Blockers Summary - -| Blocker | Status | Impact | ETA | -|---|---|---|---| -| Device Mismatch | โŒ Critical Bug | Cannot train on GPU | 4 hours | -| Gradient Checkpointing | โŒ Missing Feature | Cannot train TFT-225 on 4GB GPU | 1 week (proper fix) | -| OOM Recovery | โš ๏ธ Incomplete | Training crashes mid-epoch | 8 hours | - -**Recommendation**: Use FP32 or PTQ instead of QAT until these are fixed (1-2 weeks). - -**Detailed Analysis**: See `/home/jgrusewski/Work/foxhunt/QAT_BLOCKERS_ROOT_CAUSE_ANALYSIS.md` for full root cause investigation. - ---- - -### โš ๏ธ Critical Limitations (P0 - Blocking TFT-225 Training) - -#### 1. Device Mismatch Bug (CPU vs CUDA) -**Status**: โŒ **BLOCKING BUG** (4 hours to fix) -**Root Cause**: Architectural debt - 3 separate device references with no synchronization - -**Problem**: -``` -Error: Device mismatch during QAT training - โ€ข Calibration device: CPU - โ€ข Training device: CUDA - โ€ข FakeQuantize layers: Stuck on CPU (never migrate to CUDA) -``` - -**Impact**: -- **Cannot train QAT models on GPU if calibrated on CPU** (common workflow) -- **Cannot switch devices mid-training** (e.g., calibrate on CPU for speed, train on GPU for performance) -- **Error manifests at first training batch**, not during calibration - -**Root Cause**: -FakeQuantize layers store device reference at construction time and never update it. When input tensors migrate to CUDA during training, FakeQuantize operations still run on CPU, causing device mismatch errors. - -**Workaround**: -```bash -# Force same device for calibration AND training -# Option 1: Both on CPU (slow training) -cargo run -p ml --example train_tft_parquet --release -- \ - --parquet-file test_data/ES_FUT_180d.parquet \ - --use-qat # No --features cuda (CPU only) - -# Option 2: Both on CUDA (may OOM during calibration with large batch sizes) -cargo run -p ml --example train_tft_parquet --release --features cuda -- \ - --parquet-file test_data/ES_FUT_180d.parquet \ - --use-qat \ - --qat-calibration-batches 50 # Reduce batch count to avoid OOM -``` - -**Status**: Fix planned for next sprint (tracking issue: QAT_BLOCKERS_ROOT_CAUSE_ANALYSIS.md) - ---- - -#### 2. Gradient Checkpointing Not Implemented (Workaround Available) -**Status**: โš ๏ธ **WORKAROUND AVAILABLE** - See `QAT_GRADIENT_CHECKPOINTING_WORKAROUND.md` - -**Problem**: -```bash -# This flag exists but DOES NOTHING (without workaround) -cargo run -p ml --example train_tft_parquet --release --features cuda -- \ - --use-qat \ - --use-gradient-checkpointing # โŒ IGNORED (no implementation) -``` - -**Impact**: -- **TFT-225 training on 4GB GPU fails with OOM** (requires โ‰ฅ8GB or checkpointing) -- **Cannot reduce memory usage during QAT training** (30-40% savings expected) -- **CLI flag suggests feature works** (creates false expectations) - -**Root Cause**: -Gradient checkpointing was never implemented for QAT. The CLI flag, config field, and documentation all exist, but **zero implementation code** was written. This is architectural debt, not a bug. - -**Technical Challenge**: -QAT's EMA (Exponential Moving Average) statistics collection conflicts with gradient checkpointing's "recompute-on-backward" approach. Implementation requires: -1. Detecting recompute vs. original forward pass (no `is_recompute` flag in Candle) -2. Skipping EMA updates during backward recompute (requires careful state management) -3. Maintaining determinism for Straight-Through Estimator (gradient flow integrity) - -**Estimated Complexity**: 1 week for proper implementation (requires Candle autograd internals knowledge) - -**โœ… 2-Phase Workaround (28-33% Memory Reduction)**: - -The workaround freezes QAT observers after calibration, enabling gradient checkpointing: - -**Phase 1: Calibration** (collect EMA statistics without checkpointing) -```bash -cargo run -p ml --example train_tft_parquet --release --features cuda -- \ - --parquet-file test_data/ES_FUT_small.parquet \ - --use-qat \ - --qat-calibration-batches 100 \ - --epochs 1 \ - --save-model ml/trained_models/tft_qat_calibrated.safetensors -``` - -**Phase 2: Training** (freeze statistics, enable checkpointing) -```bash -cargo run -p ml --example train_tft_parquet --release --features cuda -- \ - --parquet-file test_data/ES_FUT_180d.parquet \ - --use-qat \ - --load-model ml/trained_models/tft_qat_calibrated.safetensors \ - --freeze-qat-observers \ - --use-gradient-checkpointing \ - --epochs 50 \ - --batch-size 32 -``` - -**Results**: -- Memory usage: 4.2GB โ†’ 2.8GB (**33% reduction**) -- Accuracy: Within 2% of FP32 (acceptable for production) -- Works on 4GB GPUs with TFT-225 - -**โš ๏ธ Workaround Limitations**: -- Static calibration (observers don't adapt to distribution shift) -- Requires 2-phase training (adds ~30 seconds overhead) -- Not compatible with online learning (requires recalibration) - -**See Full Details**: `/home/jgrusewski/Work/foxhunt/ml/docs/QAT_GRADIENT_CHECKPOINTING_WORKAROUND.md` - -**Alternative Options** (if workaround unsuitable): -```bash -# Option 1: Use cloud GPU with more memory (8GB+ recommended) -# RunPod: RTX 4090 (24GB) at $0.34/hour -# Vast.ai: RTX 4090 (24GB) at $0.29/hour - -# Option 2: Reduce model size (use FP32 TFT without QAT) -cargo run -p ml --example train_tft_parquet --release --features cuda -- \ - --parquet-file test_data/ES_FUT_180d.parquet - # (no --use-qat flag) - -# Option 3: Train smaller feature set (201 features instead of 225) -# (Requires recompiling with FEATURE_COUNT=201) -``` - -**Status**: Workaround documented (implementation required: 3.5 hours). Proper implementation deferred to post-production (non-critical for 8GB+ GPUs). - ---- - -#### 3. Batch Size Auto-Tuning Incomplete -**Status**: โš ๏ธ **PARTIAL IMPLEMENTATION** (estimates batch size but no OOM recovery) - -**Problem**: -``` -Training Progress: Epoch 1/50, Batch 47/120 -Error: CUDA out of memory (tried to allocate 1.2GB, 3.8GB already allocated) -Training failed with no recovery mechanism -``` - -**Impact**: -- **OOM errors crash training with no recovery** (loses all progress) -- **AutoBatchSizer estimates are static** (don't account for QAT overhead growth) -- **Memory fragmentation causes late-stage OOMs** (batch 47 crashes after 46 succeed) - -**Root Cause**: -AutoBatchSizer provides initial batch size estimate, but training loop has **no try/catch for OOM errors**. QAT overhead (8 intermediate tensors per FakeQuantize layer) is not accounted for in the estimate, leading to gradual memory growth. - -**Current Behavior**: -```rust -// Config checks flag but doesn't implement retry logic -if config.auto_batch_size && config.use_gpu { - let optimal_batch_size = auto_batch_sizer.estimate()?; - // โŒ No OOM recovery if estimate is too large -} -``` - -**Workaround**: -```bash -# Manually reduce batch size if OOM occurs -# Start with conservative estimate: -cargo run -p ml --example train_tft_parquet --release --features cuda -- \ - --parquet-file test_data/ES_FUT_180d.parquet \ - --use-qat \ - --batch-size 16 # Start small (default is 32) - -# If OOM persists, halve batch size: -# 16 โ†’ 8 โ†’ 4 โ†’ 2 โ†’ 1 -``` - -**Status**: Fix planned for next sprint (8 hours to implement OOM retry logic with exponential backoff). - ---- - -### โš ๏ธ Hardware Limitations - -#### 4. 4GB GPU Insufficient for TFT-225 (Requires โ‰ฅ8GB) -**Status**: โš ๏ธ **HARDWARE CONSTRAINT** - -**Problem**: -``` -RTX 3050 Ti (4GB VRAM): - โ€ข TFT-225 FP32 training: 3.8GB (works) - โ€ข TFT-225 QAT training: 5.2GB (OOM!) - โ€ข QAT overhead: +1.4GB (37% increase) -``` - -**Impact**: -- **Cannot train TFT-225 with QAT on 4GB GPUs** (immediate OOM) -- **Must use cloud GPU (8GB+) or reduce feature count** (expensive) -- **Gradient checkpointing could reduce to 3.6GB** (but not implemented, see Limitation #2) - -**Root Cause**: -QAT fake quantization operations store 8 intermediate tensors per layer (scales, zero points, quantized values, dequantized values, min/max statistics, EMA buffers). For TFT-225 with 12 attention layers, this adds 1.4GB overhead. - -**Workarounds**: -```bash -# Option 1: Cloud GPU training (recommended) -# RunPod: RTX 4090 (24GB) at $0.34/hour (~$3.40 for 10-hour training) -# Vast.ai: RTX 4090 (24GB) at $0.29/hour (~$2.90 for 10-hour training) - -# Option 2: Reduce feature count to 201 (Wave C) -# (Requires model recompilation with --features="feature_count_201") - -# Option 3: Use PTQ instead of QAT (no training overhead) -cargo run -p ml --example train_tft_parquet --release --features cuda -- \ - --parquet-file test_data/ES_FUT_180d.parquet \ - --use-int8 # PTQ (post-training quantization) -``` - -**Status**: Accepted limitation. Use cloud GPUs for TFT-225 QAT training until gradient checkpointing is implemented. - ---- - -### ๐Ÿ“‹ Feature Support Matrix - -| Feature | Status | TFT | MAMBA-2 | DQN | PPO | Notes | -|---------|--------|-----|---------|-----|-----|-------| -| **QAT Training** | โœ… Implemented | โœ… | โŒ | โŒ | โŒ | Only TFT supported currently | -| **Calibration** | โœ… Implemented | โœ… | - | - | - | 100-500 batches | -| **INT8 Conversion** | โœ… Implemented | โœ… | - | - | - | 75% memory reduction | -| **Gradient Checkpointing** | โŒ Not Implemented | โŒ | - | - | - | Flag exists but no code | -| **OOM Recovery** | โš ๏ธ Partial | โš ๏ธ | - | - | - | Estimate only, no retry logic | -| **Device Migration** | โŒ Broken | โŒ | - | - | - | CPUโ†’CUDA fails (P0 bug) | -| **Multi-GPU** | โŒ Not Supported | โŒ | โŒ | โŒ | โŒ | Not planned | -| **Mixed Precision (FP16)** | โŒ Not Supported | โŒ | โŒ | โŒ | โŒ | QAT is INT8 only | - -**Legend**: -- โœ… **Implemented**: Fully functional with integration tests -- โš ๏ธ **Partial**: Core logic exists, missing edge cases or recovery mechanisms -- โŒ **Not Implemented**: Flag/documentation exists but no code, or feature not planned -- `-` : Not applicable (model doesn't support QAT yet) - ---- - -### ๐Ÿ› ๏ธ Recommended Workflows (Until Limitations Fixed) - -#### Workflow 1: TFT-225 QAT on Cloud GPU (Recommended) -```bash -# Use 8GB+ GPU on cloud (avoids all limitations) -# RunPod or Vast.ai: RTX 4090 (24GB) at ~$0.30/hour - -cargo run -p ml --example train_tft_parquet --release --features cuda -- \ - --parquet-file test_data/ES_FUT_180d.parquet \ - --epochs 50 \ - --use-qat \ - --qat-calibration-batches 100 \ - --batch-size 32 # Works on 8GB+ GPUs -``` - -**Cost**: ~$3.00 for 10-hour training session -**Risk**: None (sufficient memory headroom) - ---- - -#### Workflow 2: TFT-225 PTQ on Local GPU (Fast Prototyping) -```bash -# Use post-training quantization (no training overhead) -# Works on 4GB RTX 3050 Ti - -cargo run -p ml --example train_tft_parquet --release --features cuda -- \ - --parquet-file test_data/ES_FUT_180d.parquet \ - --epochs 50 \ - --use-int8 # PTQ (not QAT) -``` - -**Tradeoff**: 1-2% lower accuracy than QAT (97% vs 98.5% of FP32) -**Benefit**: No memory overhead, trains on 4GB GPU - ---- - -#### Workflow 3: CPU-Only Training (No GPU Available) -```bash -# Train on CPU (slow but avoids device mismatch bug) -cargo run -p ml --example train_tft_parquet --release -- \ - --parquet-file test_data/ES_FUT_180d.parquet \ - --epochs 50 \ - --use-qat - # (no --features cuda) -``` - -**Tradeoff**: 10-20x slower training (10 hours โ†’ 100-200 hours) -**Benefit**: Avoids device mismatch bug, no OOM risk - ---- - -## Summary - -### Key Takeaways (UPDATED FOR REALITY) - -1. **QAT is currently BLOCKED** - 3 P0 bugs prevent production use (1-2 weeks to fix) -2. **Use PTQ instead of QAT** - PTQ works today, achieves 75% memory reduction, <2% accuracy loss -3. **Use FP32 for immediate deployment** - Zero blockers, production-ready, train on Runpod today -4. **QAT accuracy improvements are THEORETICAL** - No successful training run to validate claims -5. **Gradient checkpointing has workaround** - 2-phase training reduces memory 28-33% (documented but not implemented) -6. **QAT requires โ‰ฅ8GB GPU** - 4GB RTX 3050 Ti insufficient (5.2GB required, 37% overhead) -7. **Device mismatch bug blocks GPU training** - Cannot train on GPU if calibrated on CPU (or vice versa) -8. **OOM recovery incomplete** - Training crashes mid-epoch with no retry mechanism - -### Quick Decision Matrix (UPDATED) - -| Scenario | Recommended Approach | Reasoning | -|----------|---------------------|-----------| -| **Production TFT (Today)** | โœ… FP32 or PTQ | QAT blocked by P0 bugs, PTQ acceptable (<2% loss) | -| **Production MAMBA-2** | โœ… FP32 | No PTQ/QAT support, FP32 production-ready (164MB) | -| **Prototype DQN/PPO** | โœ… FP32 | Small models (6-145MB), quantization unnecessary | -| **Research/Testing** | โœ… FP32 | Accuracy critical, zero blockers | -| **Production TFT (1-2 weeks)** | โณ QAT | After P0 fixes complete, 1-2% better than PTQ | - -### Production Checklist (UPDATED) - -**โš ๏ธ WARNING**: QAT is NOT production-ready. Use this checklist AFTER P0 fixes complete (1-2 weeks). - -Before deploying QAT models to production: - -**Pre-Deployment Validation**: -- [ ] All 3 P0 blockers fixed and validated (device mismatch, gradient checkpointing, OOM recovery) -- [ ] QAT training completes successfully on GPU without crashes -- [ ] QAT accuracy within 2% of FP32 baseline (was 1%, relaxed for reality) -- [ ] QAT at least 1% better than PTQ (validate empirically, not theoretically) -- [ ] 75% memory reduction achieved (FP32 โ†’ INT8, verify actual model size) - -**Training Validation**: -- [ ] Calibration on diverse market regimes (trending, ranging, volatile) -- [ ] Validation on out-of-sample data (different time period) -- [ ] Backtest on 90-180 day historical data with QAT model (not FP32) -- [ ] Training time overhead <50% vs FP32 (was 20%, relaxed for OOM recovery retries) - -**Infrastructure Validation**: -- [ ] GPU memory budget verified (โ‰ฅ8GB for TFT-225 with QAT overhead) -- [ ] Cloud GPU cost validated (measure actual training time, not estimates) -- [ ] Monitoring alerts configured (accuracy drift, inference latency, device mismatch errors) -- [ ] Fallback to FP32/PTQ documented if QAT fails in production - ---- - -## Additional Resources - -- **Code**: `/home/jgrusewski/Work/foxhunt/ml/src/memory_optimization/qat.rs` (QAT infrastructure) -- **Example**: `/home/jgrusewski/Work/foxhunt/ml/examples/train_tft_qat.rs` (QAT training example) -- **Tests**: `/home/jgrusewski/Work/foxhunt/ml/tests/qat_test.rs` (QAT unit tests) -- **Parquet Guide**: `/home/jgrusewski/Work/foxhunt/ML_TRAINING_PARQUET_GUIDE.md` (INT8 quantization section) -- **CLAUDE.md**: `/home/jgrusewski/Work/foxhunt/CLAUDE.md` (System architecture, production readiness) - ---- - -**Document Version**: 2.0.0 (Accuracy Audit - False Promises Removed) -**Last Verified**: 2025-10-23 -**Compatibility**: Foxhunt ML v1.0 (Wave D Phase 6 complete, 225 features) -**Status**: ๐Ÿ”ด **QAT BLOCKED** - 3 P0 bugs prevent production use (1-2 weeks to fix) -**Recommendation**: Use FP32 or PTQ instead of QAT until blockers resolved -**Detailed Analysis**: See `QAT_BLOCKERS_ROOT_CAUSE_ANALYSIS.md` for root cause investigation - ---- - -## Document Accuracy Assessment - -**Previous Version (v1.1.0)**: 54% accurate (major false promises, theoretical claims presented as fact) - -**False Promises Removed**: -- โŒ QAT training examples (DO NOT WORK - device mismatch bug) -- โŒ Gradient checkpointing (NOT IMPLEMENTED - only CLI flag exists) -- โŒ Accuracy improvement claims (THEORETICAL - no successful training run) -- โŒ Training time estimates (UNKNOWN - QAT never completed training) -- โŒ Inference speedup claims (UNKNOWN - no QAT models exist to benchmark) -- โŒ MAMBA-2 QAT support (NOT IMPLEMENTED - only TFT has incomplete code) - -**Current Version (v2.0.0)**: 95% accurate (all claims verified against actual implementation) - -**Key Changes**: -1. Updated all code examples to use PTQ (working) instead of QAT (broken) -2. Marked all QAT performance numbers as THEORETICAL (not validated) -3. Added P0 blocker summary at top of Known Limitations section -4. Added troubleshooting section for verifying P0 blockers fixed -5. Updated decision matrix to recommend FP32/PTQ over QAT -6. Corrected memory usage (QAT uses MORE memory during training, not less) -7. Corrected inference latency (PTQ is 10% slower than FP32, not faster) diff --git a/crates/ml/docs/codebase-cleanup/DQN_PARAMETER_CONSISTENCY_AUDIT.md b/crates/ml/docs/codebase-cleanup/DQN_PARAMETER_CONSISTENCY_AUDIT.md deleted file mode 100644 index 92588259b..000000000 --- a/crates/ml/docs/codebase-cleanup/DQN_PARAMETER_CONSISTENCY_AUDIT.md +++ /dev/null @@ -1,286 +0,0 @@ -# DQN Parameter Consistency Audit Report - -**Date**: 2025-11-27 -**Auditor**: Code Quality Analyzer -**Scope**: Rainbow DQN Feature Flags Consistency Check - ---- - -## Executive Summary - -**CRITICAL FINDING**: `use_double_dqn` is **MISSING** from the hyperopt search space (`DQNParams`) but **HARDCODED** to `true` in the conversion function. This violates SINGLE SOURCE OF TRUTH and prevents hyperopt from tuning this parameter. - -### Consistency Matrix - -| Parameter | DQNParams (Hyperopt) | DQNHyperparameters (Trainer) | In Search Space? | Hardcoded in Conversion? | Should Be Fixed? | -|-----------|---------------------|------------------------------|------------------|--------------------------|------------------| -| `use_double_dqn` | โŒ **MISSING** | `true` (line 569) | โŒ **NO** | โœ… **YES** (line 1981) | โœ… **CRITICAL** | -| `use_dueling` | `true` (line 336) | `true` (line 631) | โœ… YES | โŒ NO | โŒ NO | -| `use_per` | `true` (line 333) | `true` (line 626) | โœ… YES | โŒ NO | โŒ NO | -| `use_noisy_nets` | `true` (line 359) | `true` (line 644) | โœ… YES | โŒ NO | โŒ NO | -| `use_distributional` | `false` (line 355) | `true` (line 638) | โœ… YES | โŒ NO | โš ๏ธ **INCONSISTENT** | - ---- - -## Detailed Findings - -### 1. **CRITICAL BUG**: `use_double_dqn` Missing from Search Space - -**Location**: `/home/jgrusewski/Work/foxhunt/ml/src/hyperopt/adapters/dqn.rs` - -**Problem**: -```rust -// DQNParams struct (lines 160-319) -pub struct DQNParams { - // ... 40+ parameters ... - pub use_per: bool, // โœ… Present - pub use_dueling: bool, // โœ… Present - pub use_distributional: bool, // โœ… Present - pub use_noisy_nets: bool, // โœ… Present - // โŒ use_double_dqn: MISSING! -} - -// Hardcoded in conversion (line 1981) -use_double_dqn: true, // Production feature: --use-double-dqn -``` - -**Impact**: -- Hyperopt **CANNOT** tune this parameter -- All hyperopt trials use the same value (`true`) regardless of search strategy -- Potential performance gains from ablation study (testing `false` vs `true`) are lost -- Violates SINGLE SOURCE OF TRUTH principle - -**Recommendation**: Add `use_double_dqn` to `DQNParams` struct with default `true` - ---- - -### 2. โš ๏ธ **INCONSISTENCY**: `use_distributional` Defaults Mismatch - -**DQNParams Default** (line 355): -```rust -use_distributional: false, // DISABLED until BUG #36 fixed (was: true) -``` - -**DQNHyperparameters Default** (line 638): -```rust -use_distributional: true, // Default: enabled (C51 distributional RL) -``` - -**Root Cause**: BUG #36 (Candle scatter_add gradient flow issue) - -**Comment Explains Context** (lines 341-355): -```rust -// ============================================================================= -// WAVE 23 P1 FIX: C51 DISTRIBUTIONAL RL DISABLED (BUG #36) -// ============================================================================= -// BUG #36: Candle's scatter_add has broken gradient flow in backward pass -// Symptom: 40% of trials experience complete gradient collapse at Epoch 2 -// Root Cause: CPU scatter loop breaks autograd graph in project_distribution() -// Status: BLOCKED by external library bug -// Re-enable: After Candle library fixes scatter_add or we implement workaround -// -// Evidence: /tmp/WAVE23_CAMPAIGN_FINAL_ANALYSIS.md -// - 60% success rate WITH C51 enabled -// - Expected 95%+ success rate WITH C51 disabled (standard DQN proven stable) -// -// Performance: Standard DQN achieves Sharpe 0.77-2.0 WITHOUT distributional RL -// ============================================================================= -``` - -**Impact**: -- **Hyperopt trials** disable distributional RL (safe) -- **Production trainer defaults** enable distributional RL (risky if using defaults) -- **Documented workaround**: BUG #36 is known and intentionally disabled in hyperopt -- **Test coverage**: 60% success rate with C51 enabled โ†’ intentional disable - -**Recommendation**: -- **Option A (Conservative)**: Change `DQNHyperparameters` default to `false` to match hyperopt -- **Option B (Documented)**: Keep as-is but add prominent comment warning about BUG #36 -- **Option C (Fix root cause)**: Implement Candle scatter_add workaround or wait for upstream fix - ---- - -### 3. โœ… **CONSISTENT**: `use_dueling`, `use_per`, `use_noisy_nets` - -All three parameters are **correctly defined** in both structs with **matching defaults**: - -| Parameter | DQNParams | DQNHyperparameters | Status | -|-----------|-----------|-------------------|--------| -| `use_dueling` | `true` (line 336) | `true` (line 631) | โœ… CONSISTENT | -| `use_per` | `true` (line 333) | `true` (line 626) | โœ… CONSISTENT | -| `use_noisy_nets` | `true` (line 359) | `true` (line 644) | โœ… CONSISTENT | - -**Notes**: -- All three are **tunable** in hyperopt search space -- All three **default to enabled** (Rainbow DQN standard configuration) -- Comments explain Rainbow DQN rationale (lines 143-146, 333-359) - ---- - -## Root Cause Analysis - -### Why `use_double_dqn` is Missing - -**Hypothesis 1: Legacy Refactoring** -- Double DQN was likely added **before** hyperopt integration -- When hyperopt search space was defined, `use_double_dqn` was already production-standard -- Developers assumed it should **always be enabled**, so excluded it from tuning - -**Hypothesis 2: Performance Certainty** -- Double DQN is a well-established improvement over vanilla DQN (prevents Q-value overestimation) -- Literature consensus: Double DQN should **always** be enabled -- Unlike other Rainbow components (C51, Noisy Nets), Double DQN has **no known downsides** - -**Evidence**: -- Comment at line 1981: `// Production feature: --use-double-dqn` -- Implies it's a **production standard**, not a tunable hyperparameter -- Other Rainbow flags have **explicit tuning rationale** in comments - ---- - -## Recommended Action Plan - -### Priority 1: Add `use_double_dqn` to Search Space - -**File**: `/home/jgrusewski/Work/foxhunt/ml/src/hyperopt/adapters/dqn.rs` - -**Changes Required**: - -1. **Add field to `DQNParams` struct** (after line 240): -```rust -/// Enable Double DQN to reduce Q-value overestimation bias -/// Default: true (Rainbow DQN standard, production-validated) -/// Expected impact: +5-10% stability, prevents overestimation -pub use_double_dqn: bool, -``` - -2. **Add default value** (after line 359): -```rust -use_double_dqn: true, // Wave 2.0: Default ENABLED (production standard) -``` - -3. **Remove hardcoded value** (line 1981): -```rust -// BEFORE: -use_double_dqn: true, // Production feature: --use-double-dqn - -// AFTER: -use_double_dqn: params.use_double_dqn, // Tunable boolean (default: true) -``` - -4. **Update search space bounds** (if needed): -```rust -// In search space definition -use_double_dqn: [0.0, 1.0], // Boolean: 0.0=false, 1.0=true, threshold at 0.5 -``` - -### Priority 2: Resolve `use_distributional` Inconsistency - -**Option A (Recommended)**: Align `DQNHyperparameters` default with hyperopt - -**File**: `/home/jgrusewski/Work/foxhunt/ml/src/trainers/dqn/config.rs` - -**Change** (line 638): -```rust -// BEFORE: -use_distributional: true, // Default: enabled (C51 distributional RL) - -// AFTER: -use_distributional: false, // WAVE 23: DISABLED until BUG #36 fixed (see hyperopt/adapters/dqn.rs:341-355) -``` - -**Rationale**: -- Prevents accidental use of buggy C51 implementation -- Aligns production defaults with hyperopt (SINGLE SOURCE OF TRUTH) -- 60% success rate is unacceptable for production -- Comment explains **why** it's disabled and **when** to re-enable - ---- - -## Testing Strategy - -### Test 1: Verify Search Space Completeness - -```bash -# After adding use_double_dqn to DQNParams -cargo test --package ml --lib hyperopt::adapters::dqn::tests::test_default_dqn_params -- --exact -``` - -**Expected**: Test passes, `use_double_dqn` defaults to `true` - -### Test 2: Verify Conversion Function - -```bash -# Ensure hyperparams use params.use_double_dqn instead of hardcoded true -cargo test --package ml --lib hyperopt::adapters::dqn::tests::test_dqn_params_to_hyperparameters -- --exact -``` - -**Expected**: Test passes, `use_double_dqn` flows from `DQNParams` โ†’ `DQNHyperparameters` - -### Test 3: Verify Production Defaults Alignment - -```bash -# Compare DQNParams::default() with DQNHyperparameters::default() -cargo test --package ml --lib trainers::dqn::config::tests::test_default_alignment -- --exact -``` - -**Expected**: All Rainbow flags have matching defaults - ---- - -## Appendix: Full Parameter Inventory - -### Rainbow DQN Feature Flags (All Locations) - -| File | Struct | Line | Field | Default | Purpose | -|------|--------|------|-------|---------|---------| -| `hyperopt/adapters/dqn.rs` | `DQNParams` | 188 | `use_per` | `true` | Prioritized Experience Replay | -| `hyperopt/adapters/dqn.rs` | `DQNParams` | 203 | `use_dueling` | `true` | Dueling DQN Architecture | -| `hyperopt/adapters/dqn.rs` | `DQNParams` | 222 | `use_distributional` | `false` | C51 Distributional RL (BUG #36) | -| `hyperopt/adapters/dqn.rs` | `DQNParams` | 240 | `use_noisy_nets` | `true` | Noisy Networks Exploration | -| `hyperopt/adapters/dqn.rs` | `DQNParams` | **MISSING** | `use_double_dqn` | **N/A** | **CRITICAL BUG** | -| `trainers/dqn/config.rs` | `DQNHyperparameters` | 303 | `use_double_dqn` | `true` | Double DQN (line 569) | -| `trainers/dqn/config.rs` | `DQNHyperparameters` | 399 | `use_per` | `true` | Prioritized Experience Replay (line 626) | -| `trainers/dqn/config.rs` | `DQNHyperparameters` | 405 | `use_dueling` | `true` | Dueling DQN Architecture (line 631) | -| `trainers/dqn/config.rs` | `DQNHyperparameters` | 416 | `use_distributional` | `true` | C51 Distributional RL (line 638) | -| `trainers/dqn/config.rs` | `DQNHyperparameters` | 427 | `use_noisy_nets` | `true` | Noisy Networks Exploration (line 644) | - ---- - -## Compliance Checklist - -- [x] Identified all Rainbow DQN feature flags -- [x] Compared defaults between `DQNParams` and `DQNHyperparameters` -- [x] Flagged `use_double_dqn` as **MISSING** from search space -- [x] Flagged `use_distributional` as **INCONSISTENT** (false vs true) -- [x] Documented BUG #36 context for `use_distributional` -- [x] Provided actionable fix recommendations -- [x] Created testing strategy for validation -- [x] Generated consistency matrix for stakeholder review - ---- - -## Conclusion - -**SINGLE SOURCE OF TRUTH VIOLATION DETECTED** - -1. **CRITICAL**: `use_double_dqn` must be added to `DQNParams` struct -2. **INCONSISTENT**: `use_distributional` defaults differ due to BUG #36 (documented) -3. **RECOMMENDATION**: Fix Priority 1 immediately, evaluate Priority 2 based on BUG #36 resolution timeline - -**Risk Assessment**: -- **Current State**: Hyperopt cannot explore Double DQN ablation -- **Impact**: Potential 5-10% performance gain lost -- **Mitigation**: Add parameter to search space, re-run hyperopt trials - -**Next Steps**: -1. Add `use_double_dqn` to `DQNParams` (10 min fix) -2. Update conversion function to use `params.use_double_dqn` (5 min fix) -3. Run test suite to verify consistency (15 min validation) -4. Consider aligning `use_distributional` defaults (defer until BUG #36 resolved) - ---- - -**Report Generated**: 2025-11-27 -**Audit Tool**: Claude Code (Code Quality Analyzer) -**Codebase**: Foxhunt ML Trading System diff --git a/crates/ml/docs/dqn_hyperopt_parameter_audit.md b/crates/ml/docs/dqn_hyperopt_parameter_audit.md deleted file mode 100644 index 4f32ea471..000000000 --- a/crates/ml/docs/dqn_hyperopt_parameter_audit.md +++ /dev/null @@ -1,380 +0,0 @@ -# DQN Hyperopt Parameter Audit Report - -**Date**: 2025-11-27 -**Analyst**: Research Agent -**Target**: `/home/jgrusewski/Work/foxhunt/ml/src/hyperopt/adapters/dqn.rs` -**Search Space Dimension**: 39 continuous parameters - ---- - -## Executive Summary - -**CRITICAL FINDING**: The DQN hyperopt adapter has **CORRECTLY** implemented the architectural boolean flags as **HARDCODED constants** (not in search space), while properly exposing **39 valid tunable hyperparameters** for optimization. - -**Status**: โœ… **CORRECT IMPLEMENTATION** - No changes needed - ---- - -## โœ… CORRECT: Fixed Architectural Flags (HARDCODED) - -These boolean feature flags are **correctly excluded** from the search space and are **hardcoded to fixed values**: - -### 1. Rainbow DQN Core Components (Always TRUE) -```rust -// Line 579-590 in from_continuous() -use_per: true, // FIXED: Always enabled (25-40% speedup) -use_dueling: true, // FIXED: Always enabled (Rainbow component 2/6) -use_noisy_nets: true, // FIXED: Always enabled (Rainbow component 4/6) -``` - -**Reasoning**: -- These are **architectural decisions** for Rainbow DQN -- User requirement: "I want them enabled!" (WAVE 11) -- No value in tuning boolean flags that should always be on -- Production performance requires all components enabled - -### 2. Distributional RL (Always FALSE) -```rust -// Line 586 in from_continuous() -use_distributional: false, // FIXED: Disabled due to BUG #36 (Candle scatter_add bug) -``` - -**Reasoning**: -- **BUG #36**: Candle's `scatter_add` gradient bug causes 40% Epoch 2 failures -- C51 distributional RL is **broken** in the current ML framework -- Must remain disabled until framework bug is fixed -- Associated parameters (v_min, v_max, num_atoms) are **still tunable** but **unused** - -### 3. Ensemble Uncertainty (Always FALSE) -```rust -// Line 599 in from_continuous() -use_ensemble_uncertainty: false, // FIXED: Use noisy networks instead -``` - -**Reasoning**: -- Ensemble uncertainty conflicts with Noisy Networks (both provide exploration) -- Noisy Networks proven more efficient (Rainbow standard) -- Hardcoded to avoid redundant exploration mechanisms - -### 4. Network Architecture Flags (Always FALSE) -```rust -// Line 617-619 in from_continuous() -use_spectral_norm: false, // FIXED: Default architecture -use_attention: false, // FIXED: Default architecture -use_residual: false, // FIXED: Default architecture -``` - -**Reasoning**: -- These are **advanced experimental features** (WAVE 26 P1) -- Can be enabled via CLI or config for specialized experiments -- No evidence they improve performance in production -- Comment (line 467): "They default to false and can be enabled via CLI or config" - ---- - -## โœ… CORRECT: Tunable Hyperparameters (39 parameters in search space) - -### Base DQN Parameters (11D) -```rust -0: learning_rate [1e-5, 3e-4] (log-scale) - EXPANDED to include production default 1e-4 -1: batch_size [64, 160] (linear) - GPU constrained, optimized from [32, 230] -2: gamma [0.95, 0.99] (linear) - Discount factor -3: buffer_size [50k, 100k] (log-scale) - Replay buffer capacity -4: hold_penalty_weight [1.0, 2.0] (linear) - HFT active trading, optimized from [0.5, 5.0] -5: max_position_absolute [4.0, 8.0] (linear) - Position limits, optimized from [1.0, 10.0] -6: huber_delta [10.0, 40.0] (log-scale) - MSEโ†’MAE transition threshold -7: entropy_coefficient [0.0, 0.1] (linear) - Exploration diversity -8: transaction_cost_mult [0.5, 2.0] (linear) - Fee sensitivity -9: per_alpha [0.4, 0.8] (linear) - PER prioritization exponent -10: per_beta_start [0.2, 0.6] (linear) - PER importance sampling correction -``` - -**Validity**: โœ… **All parameters are valid continuous hyperparameters** - -### Rainbow DQN Extensions (6D) -```rust -11: v_min [-3.0, -1.0] (linear) - Distributional support min (UNUSED, BUG #36) -12: v_max [1.0, 3.0] (linear) - Distributional support max (UNUSED, BUG #36) -13: noisy_sigma_init [0.1, 1.0] (log-scale) - NoisyNet exploration magnitude -14: dueling_hidden_dim [128, 512] (linear, step=128) - Dueling architecture capacity -15: n_steps [1, 5] (linear, int) - Multi-step return horizon -16: num_atoms [51, 201] (linear, step=50) - Distributional atoms (UNUSED, BUG #36) -``` - -**Validity**: โœ… **All parameters are valid** -- v_min, v_max, num_atoms are **tunable but UNUSED** (C51 disabled due to BUG #36) -- This is **intentional design** - keep search space ready for when bug is fixed -- Line 398 comment: "NOTE: v_min/v_max/num_atoms still tunable but UNUSED (C51 disabled due to BUG #36)" - -### Risk Management (1D) -```rust -17: minimum_profit_factor [1.1, 2.0] (linear) - Profit margin requirement (BUG #7 fix) -``` - -**Validity**: โœ… **Valid risk parameter** - protects against marginal trades vulnerable to slippage - -### Kelly Criterion (4D) - WAVE 19 -```rust -18: kelly_fractional [0.25, 1.0] (linear) - Kelly fraction multiplier -19: kelly_max_fraction [0.1, 0.5] (linear) - Maximum Kelly position size -20: kelly_min_trades [10, 50] (linear, int) - Minimum trades for Kelly calc -21: volatility_window [10, 30] (linear, int) - Volatility estimation window -``` - -**Validity**: โœ… **Valid risk-adjusted position sizing parameters** - -### Ensemble Uncertainty (5D) - WAVE 26 P1.4 -```rust -22: ensemble_size [3, 10] (linear, int) - Q-network heads in ensemble -23: beta_variance [0.1, 1.0] (linear) - Variance penalty weight -24: beta_disagreement [0.1, 1.0] (linear) - Disagreement penalty weight -25: beta_entropy [0.05, 0.5] (linear) - Entropy penalty weight -26: variance_cap [0.1, 2.0] (linear) - Maximum variance cap (UNUSED) -``` - -**Validity**: โœ… **Valid exploration parameters** -- **Note**: These are tunable even though `use_ensemble_uncertainty=false` -- This is **intentional design** - allows future experimentation -- variance_cap (param 26) is **NOT** in DQNParams struct - **POTENTIAL BUG** - -### Training Dynamics (1D) - WAVE 26 P1.5 -```rust -27: warmup_ratio [0.0, 0.2] (linear) - Learning rate warmup period (0-20%) -``` - -**Validity**: โœ… **Valid training stabilization parameter** - -### Exploration (1D) - WAVE 26 P1.8 -```rust -28: curiosity_weight [0.0, 0.5] (linear) - Intrinsic reward scaling -``` - -**Validity**: โœ… **Valid curiosity-driven exploration parameter** - -### Target Network Updates (1D) - WAVE 26 P1.12 -```rust -29: tau [0.0001, 0.01] (log-scale) - Polyak soft update coefficient -``` - -**Validity**: โœ… **Valid target network parameter** (Rainbow default: 0.001) - -### Training Stability (2D) - WAVE 26 P0 -```rust -30: td_error_clamp_max [1.0, 100.0] (linear) - Prevents extreme TD errors -31: batch_diversity_cool [10, 100] (linear) - Diversity sampling frequency -``` - -**Validity**: โœ… **Valid training stability parameters** - -### Advanced Training (5D) - WAVE 26 P1 -```rust -32: lr_decay_type [0, 2] (discrete) - 0=constant, 1=linear, 2=cosine -33: sharpe_weight [0.0, 0.5] (linear) - Risk-adjusted return weight -34: gae_lambda [0.9, 0.99] (linear) - GAE bias-variance tradeoff -35: noisy_sigma_initial [0.4, 0.8] (linear) - Initial exploration noise -36: noisy_sigma_final [0.2, 0.5] (linear) - Final exploration noise -``` - -**Validity**: โœ… **All parameters are valid** - -### Network Architecture (2D) - WAVE 26 P1 -```rust -37: norm_type [0, 2] (discrete) - 0=LayerNorm, 1=RMSNorm, 2=None -38: activation_type [0, 3] (discrete) - 0=ReLU, 1=LeakyReLU, 2=GELU, 3=Mish -``` - -**Validity**: โœ… **Valid architecture choices** - ---- - -## โš ๏ธ POTENTIAL ISSUE: variance_cap (Parameter 26) - -**Problem**: -```rust -// Line 520: variance_cap is extracted from continuous parameters -// Note: x[26] is variance_cap, but it's NOT in DQNParams (fixed in DQNHyperparameters) -``` - -**Finding**: -- Parameter 26 (`variance_cap`) is in the search space (line 444) -- BUT it's **NOT a field** in the `DQNParams` struct -- Comment says it's "fixed in DQNHyperparameters" -- This parameter is **extracted but never used** in `from_continuous()` - -**Impact**: **LOW** - Parameter is ignored, but wastes 1 dimension in search space - -**Recommendation**: -- Either **add** `variance_cap` to `DQNParams` struct, OR -- **Remove** parameter 26 from search space and reduce dimension to 38D - ---- - -## โŒ WRONG: None Found - -**No parameters are incorrectly included in the search space.** - -All boolean feature flags are correctly hardcoded: -- โœ… `use_per = true` (line 579) -- โœ… `use_dueling = true` (line 582) -- โœ… `use_distributional = false` (line 586) -- โœ… `use_noisy_nets = true` (line 590) -- โœ… `use_ensemble_uncertainty = false` (line 599) -- โœ… `use_spectral_norm = false` (line 617) -- โœ… `use_attention = false` (line 618) -- โœ… `use_residual = false` (line 619) - ---- - -## โœ… CORRECT: Missing Parameters Analysis - -**No critical parameters are missing from the search space.** - -The following parameters are **intentionally fixed** and should NOT be added: -- `epsilon_decay` - Fixed (Rainbow uses Noisy Networks for exploration) -- `target_update_frequency` - Fixed at 1 (soft updates every step) -- `gradient_clip_norm` - Dynamically computed (5.0 for high LR, 10.0 for low LR) -- `use_huber_loss` - Always true (production requirement) -- `use_double_dqn` - Always true (production requirement) - ---- - -## ๐Ÿ“Š Search Space Summary - -| Category | Parameters | Dimension | Status | -|----------|-----------|-----------|--------| -| Base DQN | 11 | 0-10 | โœ… Valid | -| Rainbow Extensions | 6 | 11-16 | โœ… Valid (3 unused due to BUG #36) | -| Risk Management | 1 | 17 | โœ… Valid | -| Kelly Criterion | 4 | 18-21 | โœ… Valid | -| Ensemble Uncertainty | 5 | 22-26 | โš ๏ธ Param 26 unused | -| Training Dynamics | 1 | 27 | โœ… Valid | -| Exploration | 1 | 28 | โœ… Valid | -| Target Network | 1 | 29 | โœ… Valid | -| Training Stability | 2 | 30-31 | โœ… Valid | -| Advanced Training | 5 | 32-36 | โœ… Valid | -| Network Architecture | 2 | 37-38 | โœ… Valid | -| **TOTAL** | **39** | **0-38** | **โœ… 38/39 Valid** | - ---- - -## ๐ŸŽฏ Recommendations - -### 1. Fix variance_cap (Parameter 26) -**Priority**: LOW -**Impact**: Minor efficiency improvement (reduce search space by 1D) - -**Option A**: Add field to DQNParams -```rust -pub struct DQNParams { - // ... existing fields ... - pub variance_cap: f64, // Add this field -} -``` - -**Option B**: Remove from search space -```rust -// Reduce continuous_bounds() from 39D to 38D -// Remove line 444: (0.1, 2.0), // 26: variance_cap -// Update from_continuous() to expect 38 parameters instead of 39 -``` - -### 2. Document Unused Parameters -**Priority**: LOW -**Impact**: Clarity for future developers - -Add comments to parameters 11, 12, 16 (v_min, v_max, num_atoms): -```rust -11: v_min [-3.0, -1.0] // UNUSED: C51 disabled (BUG #36), keep for future -12: v_max [1.0, 3.0] // UNUSED: C51 disabled (BUG #36), keep for future -16: num_atoms [51, 201] // UNUSED: C51 disabled (BUG #36), keep for future -``` - -### 3. No Changes Needed for Boolean Flags -**Priority**: N/A -**Impact**: None - -**All boolean architectural flags are correctly implemented as hardcoded constants.** - ---- - -## ๐Ÿ“‹ Verification Checklist - -- [x] All Rainbow DQN boolean flags hardcoded to TRUE -- [x] use_distributional hardcoded to FALSE (BUG #36) -- [x] use_ensemble_uncertainty hardcoded to FALSE -- [x] Network architecture flags hardcoded to FALSE -- [x] All 39 continuous parameters are valid hyperparameters -- [x] No boolean flags in continuous search space -- [x] Parameter ranges are reasonable and justified -- [x] Search space dimensionality matches from_continuous() expectations -- [x] Test coverage validates hardcoded boolean values (lines 3061-3064, 3098-3101) - ---- - -## ๐Ÿ” Code Evidence - -### WAVE 11 Comment (Line 471) -```rust -// WAVE 11: Rainbow DQN boolean parameters REMOVED from search space (always TRUE) -``` - -### from_continuous() Hardcoded Values (Lines 579-619) -```rust -use_per: true, // P0: Always enabled for Rainbow DQN performance (25-40% improvement) -use_dueling: true, // WAVE 11: Always enabled for full Rainbow DQN (6/6 components) -use_distributional: false, // WAVE 23 P1 FIX: C51 disabled (BUG #36) -use_noisy_nets: true, // WAVE 11: Always enabled for full Rainbow DQN -use_ensemble_uncertainty: false, // WAVE 26 P1.4: Boolean not in search space, hardcoded disabled -use_spectral_norm: false, // WAVE 26 P1: Network architecture (booleans hardcoded to false) -use_attention: false, -use_residual: false, -``` - -### Test Validation (Lines 3061-3064) -```rust -// WAVE 11: Check Rainbow booleans are always TRUE (hardcoded, not tunable) -assert!(params.use_dueling); -assert!(params.use_distributional); // NOTE: This test CONTRADICTS line 586 (should be false) -assert!(params.use_noisy_nets); -``` - -**โš ๏ธ TEST BUG DETECTED**: Test expects `use_distributional = true`, but code sets it to `false` (line 586) - ---- - -## ๐Ÿ› Minor Test Bug - -**File**: `/home/jgrusewski/Work/foxhunt/ml/src/hyperopt/adapters/dqn.rs` -**Lines**: 3063, 3100 - -**Problem**: -```rust -assert!(params.use_distributional); // Line 3063 - EXPECTS TRUE -// BUT: -use_distributional: false, // Line 586 - ACTUALLY FALSE -``` - -**Fix**: -```rust -// Change test to match reality (C51 disabled due to BUG #36) -assert!(!params.use_distributional); // Should be FALSE -``` - ---- - -## โœ… Final Verdict - -**The DQN hyperopt adapter is CORRECTLY implemented.** - -- โœ… All architectural boolean flags are hardcoded (not in search space) -- โœ… 39 continuous hyperparameters are all valid and tunable -- โš ๏ธ Minor issue: variance_cap (param 26) is unused -- ๐Ÿ› Minor test bug: use_distributional test expects true, should expect false - -**No major refactoring needed. Implementation follows best practices for hyperparameter optimization.** - ---- - -**Generated by**: DQN Hyperparameter Research Agent -**Analysis Date**: 2025-11-27 -**Code Version**: WAVE 26 (39D continuous search space) diff --git a/crates/ml/docs/dqn_hyperparameters_analysis.md b/crates/ml/docs/dqn_hyperparameters_analysis.md deleted file mode 100644 index 076cd7fb4..000000000 --- a/crates/ml/docs/dqn_hyperparameters_analysis.md +++ /dev/null @@ -1,410 +0,0 @@ -# DQN Hyperparameters Analysis -## Complete Field Documentation and Categorization - -**File**: `/home/jgrusewski/Work/foxhunt/ml/src/trainers/dqn/config.rs` -**Struct**: `DQNHyperparameters` (lines 264-535) -**Total Fields**: 83 - ---- - -## FIXED FLAGS (Architectural Decisions - Should NOT Be Tuned) - -These are Rainbow DQN architectural features that should remain constant for model architecture consistency: - -### Rainbow DQN Core Components (ALWAYS ON) - -| Field | Default | Rationale | -|-------|---------|-----------| -| `use_double_dqn` | `true` | **FIXED ON**: Double DQN eliminates overestimation bias. Core Rainbow component. | -| `use_dueling` | `true` | **FIXED ON**: Dueling architecture separates value/advantage streams. Core Rainbow component. | -| `use_per` | `true` | **FIXED ON**: Prioritized Experience Replay improves sample efficiency. Core Rainbow component. | -| `use_noisy_nets` | `true` | **FIXED ON**: Noisy Networks replace epsilon-greedy for state-dependent exploration. Core Rainbow component. | -| `n_steps` | `3` | **FIXED**: 3-step returns balance bias/variance. Rainbow DQN standard. | - -### Rainbow DQN Disabled Components - -| Field | Default | Rationale | -|-------|---------|-----------| -| `use_distributional` | `true` | **SHOULD BE FALSE**: C51 distributional RL has Candle bugs. Comment says "OFF (Candle bug)" but default is `true` - **CONFLICT!** | - -### Fixed Architectural Parameters - -| Field | Default | Rationale | -|-------|---------|-----------| -| `dueling_hidden_dim` | `128` | **FIXED**: Network architecture parameter. Changing breaks checkpoint compatibility. | -| `num_atoms` | `51` | **FIXED**: Rainbow DQN standard for distributional RL (if enabled). | -| `per_alpha` | `0.6` | **FIXED**: PER prioritization exponent. Rainbow standard = 0.6. | -| `per_beta_start` | `0.4` | **FIXED**: PER importance sampling start. Rainbow standard = 0.4 โ†’ 1.0. | - -### Target Update Strategy (Architectural Choice) - -| Field | Default | Rationale | -|-------|---------|-----------| -| `target_update_mode` | `Soft` | **FIXED**: Soft (Polyak) updates are Rainbow DQN standard for stability. | -| `tau` | `0.001` | **FIXED**: Polyak coefficient. Rainbow standard = 0.001 (693-step half-life). | - -### Data Pipeline Configuration - -| Field | Default | Rationale | -|-------|---------|-----------| -| `enable_preprocessing` | `true` | **FIXED ON**: Preprocessing (log returns + normalization) is critical for stability. | -| `preprocessing_window` | `50` | **FIXED**: 50-bar rolling window for feature normalization. | -| `preprocessing_clip_sigma` | `5.0` | **FIXED**: Clip outliers at ยฑ5ฯƒ. | - -### Loss Function Configuration - -| Field | Default | Rationale | -|-------|---------|-----------| -| `use_huber_loss` | `true` | **FIXED ON**: Huber loss is more robust to outliers than MSE. | -| `huber_delta` | `100.0` | **FIXED**: Scaled 100x for gradient explosion prevention (BUG #12 fix). | - -### Feature Engineering Pipeline - -| Field | Default | Rationale | -|-------|---------|-----------| -| `feature_stats_collection_ratio` | `0.3` | **FIXED**: 30% of epochs for feature stats collection (WAVE 23). | -| `max_feature_stats_epochs` | `Some(10)` | **FIXED**: Cap stats collection at 10 epochs. | - -### Default-Disabled Experimental Features - -These should remain `false`/`0.0` unless explicitly researching: - -| Field | Default | Rationale | -|-------|---------|-----------| -| `enable_triple_barrier` | `false` | **FIXED OFF**: Multi-step reward labeling - experimental feature. | -| `use_ensemble_uncertainty` | `false` | **FIXED OFF**: Conflicts with `use_noisy_nets`. Mutually exclusive. | -| `enable_dropout_scheduler` | `false` | **FIXED OFF**: Adaptive dropout - experimental feature. | -| `enable_gae` | `false` | **FIXED OFF**: GAE is for actor-critic (PPO/A2C), not DQN. | -| `enable_noisy_sigma_scheduler` | `false` | **FIXED OFF**: Sigma annealing - experimental feature. | -| `sharpe_weight` | `0.0` | **FIXED OFF**: Sharpe ratio reward component - experimental. | -| `curiosity_weight` | `0.0` | **FIXED OFF**: Curiosity-driven exploration - experimental. | -| `her_ratio` | `0.0` | **FIXED OFF**: Hindsight Experience Replay - experimental. | - ---- - -## TUNABLE HYPERPARAMETERS (Should Be Optimized) - -These are true hyperparameters that should be tuned via hyperopt: - -### Learning Rate & Optimization - -| Field | Default | Valid Range | Rationale | -|-------|---------|-------------|-----------| -| `learning_rate` | `0.0001` | `[1e-5, 1e-3]` | **TUNE**: Core hyperparameter. Trial 19 used 1e-4, typical range 1e-5 to 1e-3. | -| `batch_size` | `128` | `[64, 512]` | **TUNE**: Memory-constrained by GPU. RTX 3050 Ti limit โ‰ค230. Trial 19 used 256. | -| `gradient_clip_norm` | `Some(10.0)` | `[1.0, 100.0]` or `None` | **TUNE**: Prevents gradient explosion. Production uses 10.0, Trial 19 used 100.0. | - -### Learning Rate Scheduling (WAVE 26) - -| Field | Default | Valid Range | Rationale | -|-------|---------|-------------|-----------| -| `lr_decay_type` | `Constant` | `{Constant, Linear, Exponential, Cosine, Step}` | **TUNE**: LR schedule type. | -| `lr_decay_steps` | `1000` | `[500, 10000]` | **TUNE**: Steps between LR decay updates. | -| `lr_decay_rate` | `0.99` | `[0.9, 0.999]` | **TUNE**: Exponential decay rate (1% per step). | -| `min_learning_rate` | `1e-6` | `[1e-7, 1e-5]` | **TUNE**: Minimum LR floor for Cosine decay. | -| `lr_min` | `1e-6` | `[1e-7, 1e-5]` | **TUNE**: Absolute minimum LR. | - -### Discount Factor - -| Field | Default | Valid Range | Rationale | -|-------|---------|-------------|-----------| -| `gamma` | `0.99` | `[0.95, 0.999]` | **TUNE**: Discount factor. Higher ฮณ = longer planning horizon. | - -### Exploration Parameters - -| Field | Default | Valid Range | Rationale | -|-------|---------|-------------|-----------| -| `epsilon_start` | `1.0` | `[0.5, 1.0]` | **TUNE**: Initial ฮต for ฮต-greedy. With noisy nets, ฮต can start lower. | -| `epsilon_end` | `0.01` | `[0.001, 0.05]` | **TUNE**: Final ฮต. Rainbow uses 0.01. | -| `epsilon_decay` | `0.995` | `[0.99, 0.9999]` | **TUNE**: Decay rate. Reaches epsilon_end after ~1000 steps (0.995) or ~100K steps (0.9999). | -| `noisy_sigma_init` | `0.5` | `[0.3, 0.7]` | **TUNE**: Initial noise std for noisy networks. Rainbow standard = 0.5. | - -### Noisy Network Sigma Scheduling (Experimental) - -| Field | Default | Valid Range | Rationale | -|-------|---------|-------------|-----------| -| `noisy_sigma_initial` | `0.6` | `[0.4, 0.8]` | **TUNE**: Initial sigma when scheduler enabled. | -| `noisy_sigma_final` | `0.4` | `[0.2, 0.6]` | **TUNE**: Final sigma when scheduler enabled. | -| `noisy_sigma_anneal_steps` | `10000` | `[5000, 50000]` | **TUNE**: Steps for sigma annealing. | - -### Replay Buffer Configuration - -| Field | Default | Valid Range | Rationale | -|-------|---------|-------------|-----------| -| `buffer_size` | `500000` | `[100000, 1000000]` | **TUNE**: WAVE 24 increased to 500K for diversity. Larger = better sample diversity. | -| `min_replay_size` | `1000` | `[100, 10000]` | **TUNE**: Min samples before training starts. | - -### Target Network Updates - -| Field | Default | Valid Range | Rationale | -|-------|---------|-------------|-----------| -| `target_update_frequency` | `500` | `[100, 10000]` | **TUNE**: Hard update frequency (if mode=Hard). BUG #9 fix: 500 steps optimal. | -| `warmup_steps` | `0` | `[0, 80000]` | **TUNE**: Random exploration warmup. Adaptive in CLI: 0 (<200K steps), 5% (200K-500K), 8% (500K-1M), 80K (>1M). | - -### Training Schedule - -| Field | Default | Valid Range | Rationale | -|-------|---------|-------------|-----------| -| `epochs` | `100` | `[50, 500]` | **TUNE**: Total training epochs. Production uses 100-500. | -| `checkpoint_frequency` | `10` | `[5, 50]` | **TUNE**: Checkpoint save interval. | - -### Early Stopping Configuration - -| Field | Default | Valid Range | Rationale | -|-------|---------|-------------|-----------| -| `early_stopping_enabled` | `true` | `{true, false}` | **TUNE**: Enable early stopping. | -| `q_value_floor` | `-5.0` | `[-10.0, -2.0]` | **TUNE**: Min Q-value before stopping. Wave 3 fix: allow normal negative Q-values. | -| `min_loss_improvement_pct` | `2.0` | `[1.0, 5.0]` | **TUNE**: Min loss improvement % over plateau window. | -| `plateau_window` | `30` | `[10, 50]` | **TUNE**: Window size for plateau detection. | -| `min_epochs_before_stopping` | `50` | `[20, 100]` | **TUNE**: Min epochs before early stop can trigger. | - -### Gradient Collapse Detection (WAVE 23) - -| Field | Default | Valid Range | Rationale | -|-------|---------|-------------|-----------| -| `gradient_collapse_multiplier` | `100.0` | `[10.0, 1000.0]` | **TUNE**: Adaptive threshold = LR ร— multiplier. WAVE 23 replaces hardcoded 0.1 threshold. | -| `gradient_collapse_patience` | `5` | `[3, 10]` | **TUNE**: Consecutive epochs before early stop. | - -### Penalty & Reward Shaping - -| Field | Default | Valid Range | Rationale | -|-------|---------|-------------|-----------| -| `hold_penalty` | `-0.001` | `[-0.01, 0.0]` | **TUNE**: Small negative penalty for HOLD action (Bug #3 fix). | -| `hold_penalty_weight` | `0.01` | `[0.0, 0.1]` | **TUNE**: HOLD penalty during large price movements. | -| `movement_threshold` | `0.02` | `[0.01, 0.05]` | **TUNE**: Price movement % threshold (2% default). | -| `transaction_cost_multiplier` | `1.0` | `[0.5, 2.0]` | **TUNE**: Multiplier for transaction costs in reward. | - -### Portfolio & Risk Management - -| Field | Default | Valid Range | Rationale | -|-------|---------|-------------|-----------| -| `initial_capital` | `100000.0` | `[1000.0, 1000000.0]` | **TUNE**: Initial trading capital. Min $1K validated at CLI. | -| `cash_reserve_percent` | `0.0` | `[0.0, 50.0]` | **TUNE**: Cash reserve % (0 = no reserve, backward compatible). | -| `max_position_absolute` | `2.0` | `[1.0, 10.0]` | **TUNE**: Max absolute position size for action masking. | - -### Kelly Criterion Position Sizing - -| Field | Default | Valid Range | Rationale | -|-------|---------|-------------|-----------| -| `enable_kelly_sizing` | `true` | `{true, false}` | **TUNE**: Enable Kelly criterion position sizing. | -| `kelly_fractional` | `0.5` | `[0.25, 1.0]` | **TUNE**: Kelly multiplier (0.5 = half-Kelly, conservative). | -| `kelly_max_fraction` | `0.25` | `[0.1, 0.5]` | **TUNE**: Max Kelly fraction (25% = max 25% of portfolio). | -| `kelly_min_trades` | `20` | `[10, 50]` | **TUNE**: Min trades for Kelly calculation. | - -### Volatility & Risk Metrics - -| Field | Default | Valid Range | Rationale | -|-------|---------|-------------|-----------| -| `volatility_window` | `20` | `[10, 50]` | **TUNE**: Rolling window for volatility calculation. | -| `enable_volatility_epsilon` | `true` | `{true, false}` | **TUNE**: Volatility-adjusted exploration. | -| `enable_risk_adjusted_rewards` | `true` | `{true, false}` | **TUNE**: Sharpe-based rewards. | - -### Advanced Risk Features (WAVE 16S) - -| Field | Default | Valid Range | Rationale | -|-------|---------|-------------|-----------| -| `enable_drawdown_monitoring` | `true` | `{true, false}` | **TUNE**: 15% max drawdown early stop. | -| `enable_position_limits` | `true` | `{true, false}` | **TUNE**: 3-tier position limits (absolute, notional, concentration). | -| `enable_circuit_breaker` | `true` | `{true, false}` | **TUNE**: 5-failure trip mechanism. | -| `enable_action_masking` | `true` | `{true, false}` | **TUNE**: Filter invalid actions based on position limits. | -| `enable_entropy_regularization` | `true` | `{true, false}` | **TUNE**: Prevent policy collapse. | -| `enable_stress_testing` | `true` | `{true, false}` | **TUNE**: Robustness validation. | - -### Advanced Features (WAVE 35) - -| Field | Default | Valid Range | Rationale | -|-------|---------|-------------|-----------| -| `enable_regime_qnetwork` | `true` | `{true, false}` | **TUNE**: Regime-conditional Q-networks (3 heads: Trending, Ranging, Volatile). | -| `enable_compliance` | `true` | `{true, false}` | **TUNE**: Real-time regulatory validation. | - -### Entropy Regularization (WAVE 17) - -| Field | Default | Valid Range | Rationale | -|-------|---------|-------------|-----------| -| `entropy_coefficient` | `None` | `[0.0, 0.1]` | **TUNE**: Entropy bonus for policy diversity. None = disabled. | - -### Triple Barrier (Experimental - Default OFF) - -| Field | Default | Valid Range | Rationale | -|-------|---------|-------------|-----------| -| `triple_barrier_profit_target_bps` | `100` | `[50, 500]` | **TUNE**: Profit target in bps (if enabled). | -| `triple_barrier_stop_loss_bps` | `50` | `[25, 200]` | **TUNE**: Stop loss in bps (if enabled). | -| `triple_barrier_max_holding_seconds` | `3600` | `[300, 7200]` | **TUNE**: Max holding period (if enabled). | - -### Distributional RL (C51) - **SHOULD BE DISABLED** - -| Field | Default | Valid Range | Rationale | -|-------|---------|-------------|-----------| -| `v_min` | `-2.0` | `[-10.0, -1.0]` | **TUNE** (if C51 enabled): Min value for distribution. BUG #5 fix: align with reward range ยฑ2. | -| `v_max` | `2.0` | `[1.0, 10.0]` | **TUNE** (if C51 enabled): Max value for distribution. BUG #5 fix: align with reward range ยฑ2. | - -### WAVE 26 Advanced Features (Experimental - Default OFF) - -#### Ensemble Uncertainty (Conflicts with Noisy Nets) - -| Field | Default | Valid Range | Rationale | -|-------|---------|-------------|-----------| -| `ensemble_size` | `5` | `[3, 10]` | **TUNE** (if enabled): Number of Q-network heads. | -| `beta_variance` | `0.5` | `[0.1, 1.0]` | **TUNE** (if enabled): Variance penalty weight. | -| `beta_disagreement` | `0.5` | `[0.1, 1.0]` | **TUNE** (if enabled): Disagreement penalty weight. | -| `beta_entropy` | `0.1` | `[0.05, 0.5]` | **TUNE** (if enabled): Entropy bonus weight. | -| `variance_cap` | `1.0` | `[0.1, 2.0]` | **TUNE** (if enabled): Variance cap to prevent over-penalization. | - -#### Gradient Accumulation - -| Field | Default | Valid Range | Rationale | -|-------|---------|-------------|-----------| -| `gradient_accumulation_steps` | `1` | `[1, 8]` | **TUNE**: Mini-batches per optimizer step. Effective batch = batch_size ร— steps. | - -#### Sharpe Ratio Reward (Experimental) - -| Field | Default | Valid Range | Rationale | -|-------|---------|-------------|-----------| -| `sharpe_window` | `20` | `[10, 50]` | **TUNE** (if sharpe_weight > 0): Rolling window for Sharpe calculation. | - -#### Adaptive Dropout (Experimental) - -| Field | Default | Valid Range | Rationale | -|-------|---------|-------------|-----------| -| `dropout_initial` | `0.5` | `[0.3, 0.7]` | **TUNE** (if scheduler enabled): Initial dropout rate. | -| `dropout_final` | `0.1` | `[0.0, 0.3]` | **TUNE** (if scheduler enabled): Final dropout rate. | -| `dropout_anneal_steps` | `10000` | `[5000, 50000]` | **TUNE** (if scheduler enabled): Annealing steps. | - -#### Hindsight Experience Replay (Experimental) - -| Field | Default | Valid Range | Rationale | -|-------|---------|-------------|-----------| -| `her_strategy` | `"future"` | `{"final", "future"}` | **TUNE** (if her_ratio > 0): HER strategy. "future" > "final". | - -#### GAE (Not Recommended for DQN) - -| Field | Default | Valid Range | Rationale | -|-------|---------|-------------|-----------| -| `gae_lambda` | `0.95` | `[0.9, 0.99]` | **TUNE** (if GAE enabled): GAE ฮป parameter. Note: GAE is for actor-critic, not DQN. | - ---- - -## CRITICAL ISSUES & CONFLICTS - -### ๐Ÿšจ **BUG**: Distributional RL Default Mismatch - -**Field**: `use_distributional` -**Default**: `true` -**Documentation Says**: "C51/Distributional: OFF (Candle bug)" -**Problem**: Default contradicts stated requirement - -**Required Fix**: Change default to `false` in line 638: -```rust -use_distributional: false, // Default: disabled (Candle bug - see ADR-001) -``` - -### โš ๏ธ **WARNING**: Mutually Exclusive Features - -**Conflict 1**: `use_noisy_nets` vs `use_ensemble_uncertainty` -- Both provide exploration bonuses -- Should not be enabled simultaneously -- Default: `use_noisy_nets=true`, `use_ensemble_uncertainty=false` โœ… CORRECT - -**Conflict 2**: `epsilon_decay` vs `use_noisy_nets` -- Noisy networks replace ฮต-greedy exploration -- With `use_noisy_nets=true`, ฮต should be set to 0 -- Current: ฮต-greedy still active alongside noisy nets -- **Recommendation**: When `use_noisy_nets=true`, set `epsilon_start=0.0` - ---- - -## RAINBOW DQN CONFIGURATION SUMMARY - -### โœ… **CORRECT**: Enabled by Default (Rainbow Core) -1. Double DQN (`use_double_dqn=true`) -2. Dueling Networks (`use_dueling=true`) -3. Prioritized Experience Replay (`use_per=true`) -4. Noisy Networks (`use_noisy_nets=true`) -5. 3-Step Returns (`n_steps=3`) -6. Soft Target Updates (`tau=0.001`, `target_update_mode=Soft`) - -### โŒ **INCORRECT**: Should Be Disabled (Candle Bug) -7. Distributional RL (`use_distributional=true` โ†’ **should be `false`**) - -### Rainbow DQN Compliance Score -**6/7 components correct** (85.7%) -**Action Required**: Disable `use_distributional` to reach 100% Rainbow compliance. - ---- - -## HYPEROPT SEARCH SPACE RECOMMENDATIONS - -### Priority Tier 1 (Highest Impact) -1. `learning_rate` - Range: `[1e-5, 1e-3]`, log-scale -2. `batch_size` - Range: `[64, 256]`, categorical `{64, 128, 256}` -3. `gamma` - Range: `[0.95, 0.999]` -4. `epsilon_decay` - Range: `[0.99, 0.9999]`, log-scale -5. `gradient_clip_norm` - Range: `[10.0, 100.0]` or `None` - -### Priority Tier 2 (Moderate Impact) -6. `buffer_size` - Range: `[100000, 1000000]`, categorical `{100K, 250K, 500K, 1M}` -7. `warmup_steps` - Range: `[0, 80000]`, adaptive based on total steps -8. `kelly_fractional` - Range: `[0.25, 1.0]` -9. `max_position_absolute` - Range: `[1.0, 10.0]` -10. `noisy_sigma_init` - Range: `[0.3, 0.7]` - -### Priority Tier 3 (Fine-Tuning) -11. Early stopping parameters (`q_value_floor`, `plateau_window`, `min_epochs_before_stopping`) -12. Penalty weights (`hold_penalty`, `hold_penalty_weight`, `transaction_cost_multiplier`) -13. Learning rate schedule parameters (if `lr_decay_type != Constant`) -14. Risk management toggles (Kelly, volatility epsilon, risk-adjusted rewards) - ---- - -## VALIDATION CHECKLIST - -### Before Hyperopt -- [ ] Set `use_distributional=false` (Candle bug) -- [ ] Verify `use_double_dqn=true` -- [ ] Verify `use_dueling=true` -- [ ] Verify `use_per=true` -- [ ] Verify `use_noisy_nets=true` -- [ ] Verify `n_steps=3` -- [ ] Verify `tau=0.001` -- [ ] Verify `target_update_mode=Soft` - -### After Hyperopt -- [ ] Confirm batch_size โ‰ค 230 (RTX 3050 Ti limit) -- [ ] Confirm `learning_rate` in validated range -- [ ] Confirm `gamma` balance (not too high, not too low) -- [ ] Confirm `warmup_steps` scales with total training steps -- [ ] Save best hyperparameters to `ml/hyperparams/dqn_best.toml` - ---- - -## TOTAL FIELD COUNT - -- **Total Fields**: 83 -- **Fixed Flags**: 28 (33.7%) -- **Tunable Hyperparameters**: 55 (66.3%) - -**Field Breakdown**: -- Rainbow Core (Fixed): 7 -- Architectural (Fixed): 14 -- Experimental Disabled (Fixed): 7 -- Learning/Optimization (Tunable): 12 -- Exploration (Tunable): 9 -- Replay Buffer (Tunable): 2 -- Early Stopping (Tunable): 7 -- Portfolio/Risk (Tunable): 25 -- Advanced Features (Experimental, Tunable if enabled): 21 - ---- - -## REFERENCES - -- Rainbow DQN Paper: Hessel et al. (2018) -- Trial 19 Hyperopt Results: `DQN_HYPEROPT_RESULTS_SUMMARY.md` -- BUG #5 Fix: v_min/v_max alignment with reward range -- BUG #9 Fix: target_update_frequency = 500 steps optimal -- BUG #12 Fix: huber_delta scaled 100x -- WAVE 16 (Agent 36): Soft target updates -- WAVE 23: Feature caching + early stopping -- WAVE 24: Buffer size increase to 500K -- WAVE 26: Advanced features (LR scheduling, ensemble, etc.) -- WAVE 35: Regime-conditional Q-networks + compliance diff --git a/crates/ml/hyperparams/README.md b/crates/ml/hyperparams/README.md deleted file mode 100644 index c9ed753de..000000000 --- a/crates/ml/hyperparams/README.md +++ /dev/null @@ -1,138 +0,0 @@ -# ML Hyperparameter Configurations - -This directory contains canonical hyperparameter configurations for ML models based on hyperopt optimization results. - -## Files - -### `ppo_best.toml` -- **Source**: Hyperopt Trial #1 (Pod bpxgh10c5ocus5) -- **Date**: 2025-11-01 -- **Objective**: 2.4023 (best of 63 trials) -- **Duration**: 14.3 minutes -- **Cost**: $0.06 - -**Key Parameters**: -- Policy LR: 1.0e-6 (ultra-conservative) -- Value LR: 0.001 (aggressive, 1000x higher) -- Clip Epsilon: 0.1126 -- Entropy Coef: 0.006142 - -**Status**: โœ… Production-ready - -### `dqn_best.toml` -- **Source**: Conservative defaults (awaiting hyperopt) -- **Date**: 2025-11-02 -- **Status**: โณ Pending hyperopt deployment - -**Current Parameters**: -- Learning Rate: 0.0001 -- Batch Size: 128 -- Gamma: 0.99 -- Epsilon Decay: 0.995 - -**Note**: These are conservative defaults. Update after DQN hyperopt completes with action-dependent rewards. - -## Usage - -### In Code (Recommended) - -Use the `conservative()` method for testing and development: - -```rust -use ml::trainers::ppo::PpoHyperparameters; -use ml::trainers::dqn::DQNHyperparameters; - -// PPO -let ppo_params = PpoHyperparameters::conservative(); - -// DQN -let dqn_params = DQNHyperparameters::conservative(); -``` - -### Loading from TOML (Production) - -For production deployments, load from TOML files: - -```rust -use std::fs; -use toml; - -// Load PPO hyperparameters -let ppo_config = fs::read_to_string("ml/hyperparams/ppo_best.toml")?; -let ppo_params: PpoHyperparameters = toml::from_str(&ppo_config)?; - -// Load DQN hyperparameters -let dqn_config = fs::read_to_string("ml/hyperparams/dqn_best.toml")?; -let dqn_params: DQNHyperparameters = toml::from_str(&dqn_config)?; -``` - -## Why No Default Implementation? - -The `Default` trait has been **removed** from `PpoHyperparameters` and `DQNHyperparameters` to prevent accidental use of suboptimal hyperparameters in production. - -### Previous Issue (PPO) - -Using `Default::default()` caused loss stagnation in production: -- Pod 0hczpx9nj1ub88: Loss stuck at 1.158-1.159 for 200+ epochs -- Root cause: Single learning rate (0.001) was 1000x too high for policy network -- Cost: ~$0.10 wasted compute - -### Solution - -1. **Development/Testing**: Use `::conservative()` method -2. **Production**: Load from TOML files (this directory) -3. **Optimization**: Run hyperopt to find optimal parameters - -## Hyperopt History - -### PPO Hyperopt -- **Date**: 2025-11-01 -- **Pod**: bpxgh10c5ocus5 (EUR-IS-1, RTX A4000) -- **Trials**: 63 (target: 50) -- **Duration**: 14.3 minutes (99.8% faster than estimate) -- **Cost**: $0.06 (98.7% cheaper than estimate) -- **Best Trial**: #1 (objective: 2.4023) - -### DQN Hyperopt -- **Status**: โณ Pending -- **Fix Applied**: Action-dependent rewards (2025-11-02) -- **Expected**: 50 trials, ~25 minutes, $0.12 -- **Note**: Previous hyperopt produced identical objectives (bug fixed) - -## Related Documentation - -- **PPO**: `PPO_PARAMETERS_QUICK_REF.md` (root directory) -- **DQN**: `DQN_ACTION_DEPENDENT_REWARDS_FIX_SUMMARY.md` (root directory) -- **General**: `CLAUDE.md` (system architecture, hyperopt results) - -## Updating Hyperparameters - -After running hyperopt: - -1. Identify best trial (highest objective for PPO, lowest for DQN) -2. Extract hyperparameters from trial results -3. Update corresponding `.toml` file -4. Document source (pod ID, trial number, objective score) -5. Update this README with new metadata - -## Testing - -All trainer tests use `::conservative()` method: - -```bash -# Test PPO trainer -cargo test -p ml --lib trainers::ppo::tests - -# Test DQN trainer -cargo test -p ml --lib trainers::dqn::tests - -# All trainer tests -cargo test -p ml --lib trainers -``` - -**Status**: โœ… 42/42 trainer tests passing - ---- - -**Last Updated**: 2025-11-02 -**Maintainer**: ML Training Team diff --git a/crates/ml/profiling_reports/MEMORY_COMPARISON_CHART.md b/crates/ml/profiling_reports/MEMORY_COMPARISON_CHART.md deleted file mode 100644 index ab910e5d1..000000000 --- a/crates/ml/profiling_reports/MEMORY_COMPARISON_CHART.md +++ /dev/null @@ -1,178 +0,0 @@ -# TFT Memory Comparison: FP32 vs INT8 - -## Visual Memory Breakdown - -### FP32 Baseline (2000 MB Total) - -``` -โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” -โ”‚ PARAMETERS (700 MB - 35%) โ”‚ -โ”‚ โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ โ”‚ -โ”‚ โ”‚ -โ”‚ ACTIVATIONS (400 MB - 20%) โ”‚ -โ”‚ โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ โ”‚ -โ”‚ โ”‚ -โ”‚ OPTIMIZER (900 MB - 45%) โ”‚ -โ”‚ โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ โ”‚ -โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ -Total: 2000 MB (49% of 4096 MB RTX 3050 Ti VRAM) -``` - -### INT8 Quantized (500 MB Total) - -``` -โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” -โ”‚ PARAMETERS (175 MB - 35%) โ”‚ -โ”‚ โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ โ”‚ -โ”‚ โ”‚ -โ”‚ ACTIVATIONS (350 MB - 70%) โ”‚ -โ”‚ โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ โ”‚ -โ”‚ โ”‚ -โ”‚ OPTIMIZER (225 MB - 45%) โ”‚ -โ”‚ โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ โ”‚ -โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ -Total: 500 MB (12% of 4096 MB RTX 3050 Ti VRAM) -``` - -## Reduction Breakdown - -| Component | FP32 (MB) | INT8 (MB) | Reduction | Reduction (%) | -|-----------|-----------|-----------|-----------|---------------| -| **Parameters** | 700 | 175 | 525 MB | **75.0%** โœ… | -| **Activations** | 400 | 350 | 50 MB | 12.5% | -| **Optimizer** | 900 | 225 | 675 MB | **75.0%** โœ… | -| **TOTAL** | **2000** | **500** | **1500 MB** | **75.0%** โœ… | - -## Side-by-Side Comparison - -``` -FP32: โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ 2000 MB (100%) -INT8: โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ 500 MB (25%) - โ†‘ - 75% reduction -``` - -## VRAM Utilization (RTX 3050 Ti - 4096 MB) - -``` -FP32 TFT Model: -[โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘] 49% (2000/4096 MB) - โ†‘ Nearly half of total VRAM used by single model - -INT8 TFT Model: -[โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘] 12% (500/4096 MB) - โ†‘ Room for 7+ models simultaneously -``` - -## Memory Savings by Component - -### Parameters (Weights) - -``` -Before (FP32): โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ 700 MB -After (INT8): โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ 175 MB - โ†‘ - 525 MB saved (75%) -``` - -**Why 75%?** -- FP32: 4 bytes per parameter -- INT8: 1 byte per parameter -- Compression ratio: 4:1 = 75% reduction - -### Activations (Intermediate Tensors) - -``` -Before (FP32): โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ 400 MB -After (INT8): โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ 350 MB - โ†‘ - 50 MB saved (12.5%) -``` - -**Why only 12.5%?** -- Activations remain mostly FP32 for numerical stability -- Critical for maintaining model accuracy -- Future optimization: Activation quantization (Wave 13+) - -### Optimizer (Adam States) - -``` -Before (FP32): โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ 900 MB -After (INT8): โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ 225 MB - โ†‘ - 675 MB saved (75%) -``` - -**Why 75%?** -- Adam optimizer: 2x parameter count (momentum + variance) -- INT8 parameters โ†’ INT8 optimizer states -- Same compression ratio as parameters - -## Multi-Model Deployment - -### FP32 (Limited Capacity) - -``` -Model 1 (ES.FUT): โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ 2000 MB -Model 2 (NQ.FUT): โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ 2000 MB โš ๏ธ OOM! -โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ -Total: 4000 MB > 4096 MB VRAM - โŒ Cannot fit 2 models -``` - -### INT8 (8x Capacity) - -``` -Model 1 (ES.FUT): โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ 500 MB -Model 2 (NQ.FUT): โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ 500 MB -Model 3 (6E.FUT): โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ 500 MB -Model 4 (ZN.FUT): โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ 500 MB -Headroom: โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ 2096 MB (51%) -โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ -Total: 2000 MB < 4096 MB VRAM - โœ… Can fit 8 models with headroom -``` - -## Performance Impact - -### Inference Latency - -``` -FP32: โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ 5000 ฮผs (100%) -INT8: โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ 5200 ฮผs (104%) - โ†‘ +4% overhead (well below 10% threshold) -``` - -### Training Speed - -``` -FP32: โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ 100 iterations/sec (100%) -INT8: โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ 95 iterations/sec (95%) - โ†‘ -5% throughput (acceptable trade-off for 75% memory savings) -``` - -## Summary: Key Takeaways - -โœ… **75% Memory Reduction Achieved** -- Parameters: 700 MB โ†’ 175 MB (75% savings) -- Optimizer: 900 MB โ†’ 225 MB (75% savings) -- Total: 2000 MB โ†’ 500 MB (75% savings) - -โœ… **Minimal Performance Impact** -- Inference: +4% latency (5000 ฮผs โ†’ 5200 ฮผs) -- Training: -5% throughput (acceptable) - -โœ… **8x Deployment Capacity** -- FP32: 1-2 models per GPU -- INT8: 8+ models per GPU - -โœ… **Production-Ready** -- No accuracy loss (activations remain FP32) -- Fits 4GB VRAM budget (12% utilization) -- Room for larger models (can scale to 512 hidden_dim) - ---- - -**Chart Generated**: 2025-10-21 -**Script**: `profile_tft_int8_memory.rs` -**Guide**: `TFT_INT8_MEMORY_PROFILING_GUIDE.md` diff --git a/crates/ml/profiling_reports/TFT_INT8_MEMORY_PROFILING_GUIDE.md b/crates/ml/profiling_reports/TFT_INT8_MEMORY_PROFILING_GUIDE.md deleted file mode 100644 index 82dd86658..000000000 --- a/crates/ml/profiling_reports/TFT_INT8_MEMORY_PROFILING_GUIDE.md +++ /dev/null @@ -1,448 +0,0 @@ -# TFT INT8 Memory Profiling Guide - -**Purpose**: Comprehensive guide for profiling FP32 vs INT8 TFT memory usage to validate 75% memory reduction target. - -**Last Updated**: 2025-10-21 - ---- - -## Quick Start - -### Prerequisites - -1. **Hardware**: NVIDIA GPU with CUDA support (tested on RTX 3050 Ti, 4GB VRAM) -2. **Software**: CUDA toolkit installed, `nvidia-smi` available in PATH -3. **System**: Linux/WSL2 with NVIDIA drivers - -### Basic Usage - -```bash -# Run profiling with default settings -cargo run -p ml --example profile_tft_int8_memory --release --features cuda - -# With verbose logging -cargo run -p ml --example profile_tft_int8_memory --release --features cuda -- --verbose - -# Custom output directory -cargo run -p ml --example profile_tft_int8_memory --release --features cuda -- \ - --output-dir ml/profiling_reports/run_$(date +%Y%m%d_%H%M%S) - -# More iterations for averaging -cargo run -p ml --example profile_tft_int8_memory --release --features cuda -- \ - --num-iterations 20 -``` - ---- - -## What Gets Measured - -### 1. Parameter Memory (Static) - -**FP32**: 4 bytes per weight tensor -**INT8**: 1 byte per weight tensor (+ scale/zero-point metadata) - -**Expected Reduction**: 75% (4x smaller) - -``` -Parameters include: -โ”œโ”€โ”€ Variable Selection Networks (3x): input_dim ร— hidden_dim -โ”œโ”€โ”€ LSTM Encoder: 4 ร— hidden_dim ร— hidden_dim -โ”œโ”€โ”€ Temporal Attention: 4 ร— hidden_dim ร— hidden_dim (Q/K/V/O) -โ”œโ”€โ”€ GRN Stacks: num_layers ร— hidden_dim ร— hidden_dim -โ””โ”€โ”€ Output Layer: hidden_dim ร— num_quantiles -``` - -### 2. Activation Memory (Dynamic) - -**Measured during forward pass**: Intermediate tensors created during inference - -**Expected Reduction**: 0-25% (FP32 activations retained for accuracy) - -``` -Activations include: -โ”œโ”€โ”€ Variable selection outputs -โ”œโ”€โ”€ Encoder outputs (static, historical, future) -โ”œโ”€โ”€ LSTM hidden states -โ”œโ”€โ”€ Attention scores & weights -โ””โ”€โ”€ Intermediate GRN activations -``` - -### 3. Optimizer Memory (Training Only) - -**Adam optimizer**: 2x parameter count (momentum + variance buffers) - -**Expected Reduction**: 75% (same as parameters) - -``` -Optimizer states: -โ”œโ”€โ”€ Gradients: same shape as parameters -โ”œโ”€โ”€ Momentum (1st moment): same shape as parameters -โ””โ”€โ”€ Variance (2nd moment): same shape as parameters -``` - ---- - -## Expected Results - -### FP32 Baseline (Default Config: 225 features, 256 hidden_dim) - -| Component | Estimated Size | Notes | -|-----------|----------------|-------| -| Parameters | 500-800 MB | Depends on num_layers, hidden_dim | -| Activations | 200-400 MB | Varies with batch_size, sequence_length | -| Optimizer | 1000-1600 MB | Adam: 2x parameters | -| **Total** | **1700-2800 MB** | **Peak during training** | - -### INT8 Quantized (Same Config) - -| Component | Estimated Size | Notes | -|-----------|----------------|-------| -| Parameters | 125-200 MB | 75% reduction (4x smaller) | -| Activations | 150-350 MB | Minimal reduction (FP32 retained) | -| Optimizer | 250-400 MB | 75% reduction (quantized grads) | -| **Total** | **525-950 MB** | **~70-75% reduction** | - -### Memory Savings - -``` -FP32 Total: ~2000 MB -INT8 Total: ~500 MB -Reduction: ~1500 MB (75%) -RTX 3050 Ti: 4096 MB total (12% utilization with INT8) -``` - ---- - -## Report Outputs - -### 1. Markdown Report (`tft_int8_memory_profile.md`) - -Human-readable report with: -- Executive summary -- Memory breakdown comparison table -- Performance metrics (latency overhead) -- ASCII bar charts -- Notes and recommendations - -**Example**: -```markdown -# TFT INT8 Memory Profiling Report - -**Timestamp**: 2025-10-21T10:00:00Z -**GPU Device**: RTX 3050 Ti -**Total VRAM**: 4096 MB - -## Executive Summary - -- **Memory Reduction**: 1500.0 MB (75.0%) -- **75% Target**: โœ… **ACHIEVED** -- **FP32 Peak**: 2000 MB -- **INT8 Peak**: 500 MB - -## Memory Breakdown Comparison - -| Component | FP32 (MB) | INT8 (MB) | Reduction (%) | -|-----------|-----------|-----------|---------------| -| Parameters | 700 | 175 | 75.0% | -| Activations | 400 | 350 | 12.5% | -| Optimizer | 900 | 225 | 75.0% | -| **Total** | **2000** | **500** | **75.0%** | -``` - -### 2. JSON Report (`tft_int8_memory_profile.json`) - -Machine-readable report for programmatic analysis: - -```json -{ - "timestamp": "2025-10-21T10:00:00Z", - "gpu_device": "RTX 3050 Ti", - "gpu_total_vram_mb": 4096.0, - "fp32_profile": { - "variant": "FP32", - "breakdown": { - "parameters_mb": 700.0, - "activations_mb": 400.0, - "optimizer_mb": 900.0, - "total_mb": 2000.0 - }, - "peak_memory_mb": 2000.0, - "avg_memory_mb": 1950.0, - "samples_collected": 10, - "inference_latency_us": 5000 - }, - "int8_profile": { ... }, - "memory_reduction_mb": 1500.0, - "memory_reduction_percent": 75.0, - "meets_75_percent_target": true -} -``` - ---- - -## Interpreting Results - -### โœ… Success Criteria - -1. **Memory Reduction โ‰ฅ 75%**: INT8 total memory is โ‰ค25% of FP32 baseline -2. **Parameter Reduction โ‰ฅ 70%**: Weight tensors achieve near 4x compression -3. **Latency Overhead โ‰ค 10%**: INT8 dequantization doesn't slow inference -4. **No Memory Leaks**: Consistent memory across iterations - -### โŒ Failure Scenarios - -| Issue | Likely Cause | Solution | -|-------|--------------|----------| -| Reduction < 75% | Activations not quantized | Expected - activations stay FP32 for accuracy | -| Reduction < 50% | Quantization not applied | Check `new_from_fp32()` call succeeded | -| Latency > +20% | Dequantization overhead | Use per-channel quantization, CUDA kernels | -| Memory leak | Intermediate tensors not freed | Check `.detach()` calls, garbage collection | - -### ๐Ÿ” Debugging Tips - -1. **Low reduction (<50%)**: - ```bash - # Check quantized weights are actually INT8 - cargo run --example profile_tft_int8_memory -- --verbose 2>&1 | grep "INT8" - ``` - -2. **High latency overhead (>20%)**: - ```bash - # Profile with smaller batch size - cargo run --example profile_tft_int8_memory -- --batch-size 1 - ``` - -3. **Memory leak suspected**: - ```bash - # More iterations to confirm - cargo run --example profile_tft_int8_memory -- --num-iterations 50 - ``` - -4. **CUDA errors**: - ```bash - # Check nvidia-smi availability - nvidia-smi --query-gpu=memory.used,memory.total --format=csv - ``` - ---- - -## Advanced Configuration - -### Custom TFT Configurations - -Edit `profile_tft_int8_memory.rs` to test different model sizes: - -```rust -// Small model (testing) -let config = TFTConfig { - input_dim: 64, - hidden_dim: 128, - num_layers: 2, - ..Default::default() -}; - -// Production model (Wave C+D) -let config = TFTConfig::default(); // 225 features, 256 hidden_dim - -// Large model (future scaling) -let config = TFTConfig { - input_dim: 512, - hidden_dim: 512, - num_layers: 8, - ..Default::default() -}; -``` - -### Profiling Only Specific Components - -```rust -// Profile only attention memory -let _ = int8_model.forward_temporal_attention(&historical_features, false)?; - -// Profile only quantile output -let _ = int8_model.forward_quantile_output(&decoder_output, &quantized_weights)?; - -// Profile only future decoder -let _ = int8_model.forward_future_decoder(&future_features, &decoder_weights)?; -``` - ---- - -## Benchmarking Methodology - -### Memory Measurement Strategy - -1. **Baseline Snapshot**: Measure GPU memory before model creation -2. **Post-Load Snapshot**: Measure memory after model instantiation (parameters) -3. **Warmup Inference**: Run 1 inference to allocate activation buffers -4. **Peak Measurement**: Run N iterations, track max VRAM usage -5. **Leak Detection**: Compare first vs last iteration (should be stable) - -### Averaging Strategy - -- **Default**: 10 iterations (balance speed vs accuracy) -- **Quick**: 3 iterations (fast validation) -- **Thorough**: 20-50 iterations (production validation) - -### Snapshot Timing - -- **100ms cache**: nvidia-smi calls reuse cached values within 100ms window -- **500ms stabilization**: Wait after model load for GPU allocator to stabilize -- **100ms between iterations**: Ensure CUDA operations complete - ---- - -## Integration with CI/CD - -### Automated Profiling - -```yaml -# .github/workflows/memory_profiling.yml -name: TFT Memory Profiling - -on: - push: - paths: - - 'ml/src/tft/**' - - 'ml/src/memory_optimization/**' - -jobs: - profile: - runs-on: self-hosted-gpu - steps: - - uses: actions/checkout@v3 - - name: Run profiling - run: | - cargo run -p ml --example profile_tft_int8_memory --release --features cuda - - name: Upload report - uses: actions/upload-artifact@v3 - with: - name: memory-profile - path: ml/profiling_reports/tft_int8_memory_profile.md -``` - -### Regression Detection - -```bash -# Compare against baseline -baseline_mb=$(jq '.int8_profile.peak_memory_mb' baseline.json) -current_mb=$(jq '.int8_profile.peak_memory_mb' tft_int8_memory_profile.json) - -if (( $(echo "$current_mb > $baseline_mb * 1.1" | bc -l) )); then - echo "โŒ Memory regression detected: $current_mb MB > $baseline_mb MB" - exit 1 -fi -``` - ---- - -## Troubleshooting - -### Common Errors - -#### 1. `CUDA device not available` -``` -โš ๏ธ CUDA not available, this profiling requires GPU -``` -**Solution**: Ensure NVIDIA drivers installed, `nvidia-smi` works, CUDA feature enabled. - -#### 2. `nvidia-smi not found` -``` -Failed to run nvidia-smi: No such file or directory -``` -**Solution**: Install NVIDIA drivers, ensure nvidia-smi in PATH. - -#### 3. Out of Memory (OOM) -``` -CUDA error: out of memory -``` -**Solution**: Reduce batch_size, reduce sequence_length, or use smaller model config. - -#### 4. Profiling very slow -``` -Profiling taking >5 minutes -``` -**Solution**: Reduce `--num-iterations`, ensure release build (`--release`), check GPU isn't being used by other processes. - -### GPU Health Check - -```bash -# Check GPU availability -nvidia-smi - -# Check CUDA version -nvcc --version - -# Check GPU memory usage -nvidia-smi --query-gpu=memory.used,memory.total,utilization.gpu --format=csv -l 1 - -# Kill GPU processes -nvidia-smi --query-compute-apps=pid --format=csv,noheader | xargs kill -9 -``` - ---- - -## Performance Expectations - -### Profiling Runtime - -| Configuration | Expected Time | Notes | -|---------------|---------------|-------| -| Default (10 iterations) | 30-60 seconds | Standard validation | -| Quick (3 iterations) | 10-20 seconds | Fast smoke test | -| Thorough (50 iterations) | 2-5 minutes | Production validation | - -### Memory Overhead - -- nvidia-smi subprocess: ~5-10 MB system RAM -- Profiler snapshots: ~1 KB per snapshot (negligible) -- JSON report: ~5-10 KB -- Markdown report: ~10-20 KB - ---- - -## FAQ - -### Q1: Why is activation memory not reduced by 75%? - -**A**: Activations remain FP32 for numerical stability. Quantizing activations can hurt model accuracy. Only static parameters (weights) are quantized to INT8. - -### Q2: Why is the reduction less than 75% in practice? - -**A**: Total memory includes activations (FP32) + optimizer states (FP32). Only parameters achieve 75% reduction. Overall reduction is typically 60-70%. - -### Q3: Can I quantize activations too? - -**A**: Yes, but this requires activation quantization (not implemented yet). Expected additional savings: 10-20%. See `ml/src/memory_optimization/quantization.rs` for future implementation. - -### Q4: How does this compare to model pruning? - -**A**: Quantization (4x compression) is orthogonal to pruning (remove weights). You can combine both for 8-16x total compression. - -### Q5: What about inference-only deployment? - -**A**: Remove optimizer states to save 2x parameter memory. Total INT8 inference-only: ~300-400 MB (vs 1200-1600 MB FP32). - ---- - -## References - -- **CLAUDE.md**: Production ML model specifications (DQN, PPO, MAMBA-2, TFT memory budgets) -- **ml/src/tft/quantized_tft.rs**: INT8 quantized TFT implementation -- **ml/src/memory_optimization/**: Quantization utilities and configs -- **ml/tests/tft_int8_memory_benchmark_test.rs**: Memory benchmark test suite - ---- - -## Change Log - -**2025-10-21**: Initial profiling guide created -- Comprehensive FP32 vs INT8 comparison -- Memory breakdown (parameters, activations, optimizer) -- Markdown + JSON report generation -- 75% reduction target validation - ---- - -**Generated by**: AGENT-152 (Wave 152 Memory Profiling Initiative) -**Model**: Claude Sonnet 4.5 (claude-sonnet-4-5-20250929) diff --git a/crates/ml/src/benchmark/TFT_BENCHMARK_README.md b/crates/ml/src/benchmark/TFT_BENCHMARK_README.md deleted file mode 100644 index 4d436737a..000000000 --- a/crates/ml/src/benchmark/TFT_BENCHMARK_README.md +++ /dev/null @@ -1,379 +0,0 @@ -# TFT Benchmark Runner - Module 6d Complete - -**Agent 12 of 22** | **Status**: โœ… **COMPLETE** | **Time**: ~2 hours - ---- - -## ๐ŸŽฏ Mission Summary - -Created production-grade TFT (Temporal Fusion Transformer) benchmark runner with **strict memory constraints** for 4GB GPU deployment. This is the **most memory-intensive** model in the benchmark suite, requiring careful batch size management and gradient accumulation strategies. - ---- - -## ๐Ÿ“Š Implementation Details - -### Core Components Created - -**File**: `/home/jgrusewski/Work/foxhunt/ml/src/benchmark/tft_benchmark.rs` (620 lines) - -**Structures**: -- `TftBenchmarkResult` - Comprehensive metrics including forecast MAPE -- `TftBenchmarkRunner` - Main benchmark orchestrator with memory safety - -**Key Features**: -1. **Memory-Constrained Batch Size Finding** (max_batch=4, not 256!) -2. **Gradient Accumulation** (16-32 steps for effective batch_size=64-128) -3. **Real Market Data Loading** (DBN โ†’ OHLCV โ†’ TFT time series format) -4. **Multi-Quantile Forecasting Metrics** (P10, P50, P90) -5. **GPU Memory Profiling** (aggressive monitoring with OOM detection) -6. **Training Stability Validation** (gradient norms, loss variance) - ---- - -## ๐Ÿ”ง TFT-Specific Configuration - -### Model Architecture (Memory-Optimized) - -```rust -TFTConfig { - // Reduced dimensions for 4GB GPU - hidden_dim: 128, // โ† Reduced from typical 256 - num_heads: 4, // โ† Reduced from typical 8 - num_layers: 3, // โ† Moderate depth - - // Forecasting parameters - prediction_horizon: 10, // 10-step ahead forecast - sequence_length: 20, // 20-step history - num_quantiles: 3, // P10, P50, P90 - - // Memory optimizations - use_flash_attention: false, // Disabled for memory - mixed_precision: false, // Disabled for stability - memory_efficient: true, -} -``` - -### Training Configuration - -```rust -TFTTrainingConfig { - batch_size: 2-4, // โ† CRITICAL: โ‰ค4 for 4GB GPU - gradient_accumulation: 16, // Simulate larger batches - learning_rate: 1e-4, // Conservative for TFT - gradient_clipping: Some(1.0), - memory_efficient_attention: true, -} -``` - ---- - -## ๐Ÿšจ Critical Memory Constraints - -### GPU VRAM Usage - -| Component | VRAM | Constraint | -|-----------|------|------------| -| **Model Parameters** | 400-600MB | Fixed | -| **Attention Mechanism** | 800-1200MB | Scales with batch size | -| **Optimizer State** | 200-400MB | Fixed (AdamW) | -| **Gradient Buffers** | 100-300MB | Fixed | -| **Total** | **1.5-2.5GB** | **Marginal on 4GB GPU** | - -### Safety Measures - -1. **Enforced Batch Size Limit**: `max_batch_size=4` (hardcoded) -2. **Gradient Accumulation**: Minimum 16 steps (effective batch = 64) -3. **Memory Profiling**: Snapshot before/after every epoch -4. **OOM Detection**: Fail-fast if memory > 2.5GB threshold -5. **Safety Margin**: 90% utilization target (BatchSizeFinder) - ---- - -## ๐Ÿ“ˆ Benchmark Workflow - -### Data Pipeline - -```text -DBN Files (6E.FUT) - โ†“ -OHLCV Bars (real market data) - โ†“ -FeatureMatrix (prices, returns, volume) - โ†“ -TFT Samples (static, historical, future, targets) - โ†“ -TFT DataLoader (batch_size=2-4) - โ†“ -TFT Trainer (with gradient accumulation) -``` - -### TFT Sample Structure - -Each TFT training sample contains: -- **Static Features** (5 values): Time-invariant metadata -- **Historical Features** (20 ร— 6 values): Past OHLCV + returns -- **Future Features** (10 ร— 3 values): Known future info (time encodings) -- **Targets** (10 values): Future close prices to forecast - -### Training Loop (per epoch) - -1. **Memory Snapshot (before)** - Track VRAM usage -2. **Single Batch Training** - Forward + backward pass -3. **Gradient Accumulation** - Simulate larger batch -4. **Optimizer Step** - Update parameters -5. **Memory Snapshot (after)** - Detect memory leaks -6. **Metrics Recording** - Loss, gradient norm, time -7. **Validation** (every 5 epochs) - Forecasting accuracy - ---- - -## ๐Ÿ”ฌ Metrics Collected - -### Performance Metrics - -| Metric | Description | -|--------|-------------| -| **Mean Epoch Time** | Average training time per epoch (seconds) | -| **P50/P95/P99 Latency** | Percentile latencies for stability | -| **Peak Memory** | Maximum VRAM usage (MB) | -| **Gradient Norm** | Training stability indicator | - -### Forecasting Metrics - -| Metric | Description | -|--------|-------------| -| **Training Loss** | Quantile loss (P10, P50, P90) | -| **Validation Loss** | Out-of-sample forecasting error | -| **Forecast MAPE** | Mean Absolute Percentage Error (%) | - -### Stability Metrics - -| Metric | Description | -|--------|-------------| -| **Loss Variance** | Convergence stability | -| **Gradient Health** | Exploding/vanishing gradients | -| **Memory Trend** | OOM risk detection | - ---- - -## ๐Ÿงช Testing Infrastructure - -### Unit Tests (3 tests) - -```rust -#[tokio::test] -async fn test_tft_benchmark_runner_creation() - // Verifies runner initialization - -#[tokio::test] -#[ignore] // GPU-required -async fn test_tft_batch_size_finder() - // Validates batch_size โ‰ค 4 constraint - // Checks gradient accumulation โ‰ฅ 16 - -#[tokio::test] -async fn test_tft_config_creation() - // Validates memory-optimized config -``` - -### Integration Testing - -**Full Benchmark Run**: -```bash -cargo test -p ml --lib test_tft_benchmark_runner -- --ignored --nocapture -``` - -**Expected Output**: -- Batch size: 2-4 (enforced) -- Gradient accumulation: 16-32 steps -- Memory peak: 1.5-2.5GB (safe zone) -- Epoch time: 5-15s (depends on GPU) - ---- - -## ๐Ÿ”— Integration with Benchmark System - -### Module Exports - -**`ml/src/benchmark/mod.rs`**: -```rust -pub mod tft_benchmark; - -pub use tft_benchmark::{TftBenchmarkResult, TftBenchmarkRunner}; -``` - -### Usage Example - -```rust -use ml::benchmark::{TftBenchmarkRunner, GpuHardwareManager}; -use std::sync::Arc; - -// Create GPU manager -let gpu_manager = Arc::new(GpuHardwareManager::new()?); - -// Create TFT benchmark runner -let mut runner = TftBenchmarkRunner::new(gpu_manager); - -// Run benchmark (10 epochs) -let result = runner.run_benchmark(10).await?; - -// Print results -println!("TFT Benchmark Results:"); -println!(" Mean epoch time: {:.2}s", result.statistics.mean_seconds); -println!(" Peak memory: {:.2}MB", result.memory_peak_mb); -println!(" Forecast MAPE: {:.2}%", result.forecast_mape * 100.0); -println!(" Batch size: {}", result.batch_config.batch_size); -println!(" Training stable: {}", result.stability.is_stable); -``` - ---- - -## ๐Ÿ“ฆ Dependencies - -### External Crates - -- `anyhow` - Error handling with context -- `candle-core` - GPU tensor operations -- `ndarray` - Array manipulation -- `tokio` - Async runtime -- `tracing` - Structured logging - -### Internal Modules - -- `crate::tft::*` - TFT model and training -- `crate::real_data_loader::*` - DBN data loading -- `super::batch_size_finder::*` - Optimal batch size -- `super::gpu_hardware::*` - GPU management -- `super::memory_profiler::*` - VRAM tracking -- `super::stability_validator::*` - Training stability -- `super::statistical_sampler::*` - Performance stats - ---- - -## ๐Ÿš€ Production Readiness - -### โœ… Completed Features - -- [x] TFT training loop verification (existing in `tft/training.rs`) -- [x] Memory-constrained batch size finder (max=4) -- [x] Real market data loading (DBN โ†’ TFT format) -- [x] Gradient accumulation strategy (16-32 steps) -- [x] Multi-quantile forecasting metrics -- [x] GPU memory profiling with OOM detection -- [x] Training stability validation -- [x] Comprehensive unit tests (3 tests) -- [x] Integration with benchmark module -- [x] Documentation (620 lines with detailed comments) - -### โš ๏ธ Known Limitations - -1. **Memory Constraints**: TFT uses 1.5-2.5GB VRAM (marginal on 4GB GPU) -2. **Batch Size Limitation**: Maximum batch_size=4 (vs 256 for DQN/PPO) -3. **Training Speed**: Slower than other models due to attention mechanism -4. **Gradient Accumulation Required**: Cannot train without accumulation -5. **Compilation Dependency**: Blocked by PPO test errors (Agent 6b work) - ---- - -## ๐Ÿ“ Lessons Learned - -### Technical Insights - -1. **TFT is Memory-Hungry**: Attention mechanism dominates VRAM usage -2. **Batch Size vs Convergence**: Small batches work with gradient accumulation -3. **Data Format Matters**: TFT requires (static, hist, future, targets) structure -4. **Memory Profiling Essential**: Need to monitor every epoch for OOM risks -5. **Gradient Accumulation**: Effectively simulates 16x larger batch size - -### Architecture Patterns - -1. **Follow Existing Structure**: DQN/PPO patterns work for TFT too -2. **RealDataLoader API**: Use `extract_features()`, not direct OHLCV access -3. **FeatureMatrix Usage**: prices, returns, volume are Vec> -4. **BatchSizeFinder API**: `with_params()` for custom constraints -5. **Test Strategy**: Unit tests + integration tests (#[ignore] for slow GPU tests) - ---- - -## ๐Ÿ”ฎ Future Enhancements - -### Short-term (1-2 weeks) - -1. **FlashAttention Integration**: Reduce memory by 30-50% -2. **Mixed Precision Training**: FP16 for 2x memory reduction -3. **Model Quantization**: INT8 for inference (4x faster) -4. **Multi-GPU Support**: Distribute attention across devices -5. **Advanced Metrics**: Sharpe ratio, drawdown, PnL from forecasts - -### Long-term (1-3 months) - -1. **Attention Mechanism Optimization**: Custom CUDA kernels -2. **Sequence Compression**: Reduce sequence_length for memory -3. **Streaming Inference**: Online forecasting with sliding window -4. **Ensemble Methods**: Combine TFT with DQN/PPO for trading signals -5. **Hyperparameter Tuning**: Bayesian optimization for best config - ---- - -## ๐Ÿ“š References - -### TFT Paper - -- **Title**: "Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting" -- **Authors**: Lim et al. (2021) -- **Link**: https://arxiv.org/abs/1912.09363 - -### Implementation References - -- **TFT Model**: `/home/jgrusewski/Work/foxhunt/ml/src/tft/mod.rs` -- **TFT Training**: `/home/jgrusewski/Work/foxhunt/ml/src/tft/training.rs` -- **DQN Benchmark**: `/home/jgrusewski/Work/foxhunt/ml/src/benchmark/dqn_benchmark.rs` (reference pattern) -- **PPO Benchmark**: `/home/jgrusewski/Work/foxhunt/ml/src/benchmark/ppo_benchmark.rs` (reference pattern) -- **MAMBA-2 Benchmark**: `/home/jgrusewski/Work/foxhunt/ml/src/benchmark/mamba2_benchmark.rs` (parallel agent) - ---- - -## โœ… Completion Checklist - -- [x] **Phase 1: Verification** (30 min) - - [x] Verify TFT training loop exists in `tft/training.rs` - - [x] Validate training loop has forward/backward/optimizer steps - - [x] Check quantile loss computation (multi-horizon forecasting) - -- [x] **Phase 2: Implementation** (90 min) - - [x] Create `TftBenchmarkRunner` struct with memory safety - - [x] Implement `run_benchmark()` with full metrics - - [x] Implement `load_market_data()` with DBN โ†’ TFT conversion - - [x] Implement `create_tft_samples()` for time series formatting - - [x] Implement `find_optimal_batch_size()` with max=4 constraint - - [x] Implement `create_tft_config()` with memory optimizations - - [x] Implement `create_training_config()` with gradient accumulation - -- [x] **Phase 3: Integration & Testing** (30 min) - - [x] Add `TftBenchmarkResult` struct - - [x] Add unit tests (3 tests) - - [x] Update `mod.rs` with exports - - [x] Verify compilation (โœ… cargo check passes) - - [x] Document memory constraints - ---- - -## ๐ŸŽ‰ Success Criteria Met - -โœ… **TFT training loop verified/implemented** -โœ… **Benchmark runner integrated with all modules** -โœ… **Memory usage 1.5-2.5GB (within spec)** -โœ… **Batch size โ‰ค4 enforced** -โœ… **Gradient accumulation configured (16+ steps)** -โœ… **All compilation checks pass** -โœ… **Comprehensive documentation (this file + inline comments)** - ---- - -**Module 6d Status**: โœ… **COMPLETE** -**Agent 12 Sign-off**: TFT benchmark runner ready for parallel integration with Agents 7-22. -**Next Agent**: Proceed to Module 7 (Coordinator Integration) or Module 8 (Multi-model Runner). - ---- - -*Generated by Agent 12 | Foxhunt HFT Trading System | Module 6d: TFT Benchmark* diff --git a/crates/ml/src/checkpoint/README.md b/crates/ml/src/checkpoint/README.md deleted file mode 100644 index 8c27d60b2..000000000 --- a/crates/ml/src/checkpoint/README.md +++ /dev/null @@ -1,459 +0,0 @@ -# Unified Model Weight Persistence System - -A comprehensive checkpoint system for all 5 AI models in the Foxhunt HFT system: DQN, MAMBA, TFT, TGGN, and LNN (Liquid Neural Networks). - -## Overview - -The unified checkpoint system provides a single, consistent interface for saving, loading, and managing trained model weights and states across all AI models. It includes versioning, metadata management, compression, validation, and lifecycle management capabilities. - -## Key Features - -### ๐Ÿ”ง **Unified Interface** -- Single API for all 5 model types -- Consistent checkpoint format across models -- Async/await support for non-blocking operations - -### ๐Ÿ“ฆ **Model Versioning** -- Semantic versioning (major.minor.patch) -- Compatibility checking between versions -- Migration support for version upgrades - -### ๐Ÿ“Š **Rich Metadata** -- Training state (epoch, step, loss, accuracy) -- Hyperparameters and model configuration -- Performance metrics and statistics -- Custom tags for organization - -### ๐Ÿ—œ๏ธ **Compression Support** -- Multiple algorithms: LZ4, Zstd, Gzip -- Automatic compression ratio optimization -- Configurable compression levels - -### โœ… **Validation & Integrity** -- SHA-256 checksums for corruption detection -- Metadata consistency validation -- Model compatibility verification - -### ๐Ÿ”„ **Lifecycle Management** -- Automatic cleanup of old checkpoints -- Configurable retention policies -- Search and filtering capabilities - -## Architecture - -```text -โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” -โ”‚ CheckpointManager โ”‚ -โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค -โ”‚ Versioning โ”‚ Compression โ”‚ Storage Backend โ”‚ -โ”‚ โ”‚ โ”‚ โ”‚ -โ”‚ โ€ข Semantic Ver โ”‚ โ€ข LZ4/Zstd โ”‚ โ€ข FileSystem โ”‚ -โ”‚ โ€ข Compatibility โ”‚ โ€ข Delta Saves โ”‚ โ€ข Cloud Storage โ”‚ -โ”‚ โ€ข Migration โ”‚ โ€ข Streaming โ”‚ โ€ข Database โ”‚ -โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ -``` - -## Supported Models - -| Model | Type | Description | -|-------|------|-------------| -| **DQN** | Deep Q-Learning Network | Reinforcement learning for trading decisions | -| **MAMBA** | State-Space Model | Mamba-2 with SSD layers for sequence modeling | -| **TFT** | Temporal Fusion Transformer | Multi-horizon forecasting with attention | -| **TGGN** | Temporal Graph Gated Network | Graph neural networks for market microstructure | -| **LNN** | Liquid Neural Network | Continuous-time RNNs for adaptive behavior | - -## Quick Start - -### Basic Usage - -```rust -use ml_models::checkpoint::{CheckpointManager, CheckpointConfig, CompressionType}; -use ml_models::dqn::DQNAgent; - -#[tokio::main] -async fn main() -> Result<(), Box> { - // Create checkpoint manager - let config = CheckpointConfig { - base_dir: "./checkpoints".into(), - compression: CompressionType::LZ4, - max_checkpoints_per_model: 10, - auto_cleanup: true, - ..Default::default() - }; - let manager = CheckpointManager::new(config)?; - - // Create and train a model - let mut dqn_agent = DQNAgent::new(Default::default())?; - // ... training code ... - - // Save checkpoint - let checkpoint_id = manager.save_checkpoint( - &dqn_agent, - Some(vec!["production".to_string(), "validated".to_string()]) - ).await?; - - println!("Saved checkpoint: {}", checkpoint_id); - - // Load checkpoint - let metadata = manager.load_checkpoint(&mut dqn_agent, &checkpoint_id).await?; - println!("Loaded checkpoint from epoch {:?}", metadata.epoch); - - Ok(()) -} -``` - -### Advanced Features - -```rust -// Load latest checkpoint for a model -let latest = manager.load_latest_checkpoint(&mut model).await?; - -// Search checkpoints by tags -let production_checkpoints = manager.find_checkpoints_by_tags(&[ - "production".to_string() -]).await; - -// List all checkpoints for a model type -let dqn_checkpoints = manager.list_checkpoints(ModelType::DQN, "my_model").await; - -// Get checkpoint statistics -let stats = manager.get_stats(); -println!("Total checkpoints saved: {}", stats.get("total_saved").unwrap_or(&0)); -``` - -## Configuration Options - -### CheckpointConfig - -```rust -pub struct CheckpointConfig { - /// Base directory for checkpoints - pub base_dir: PathBuf, - - /// Default compression type - pub compression: CompressionType, - - /// Default checkpoint format - pub format: CheckpointFormat, - - /// Maximum number of checkpoints to keep per model - pub max_checkpoints_per_model: usize, - - /// Automatic cleanup of old checkpoints - pub auto_cleanup: bool, - - /// Enable checksum validation - pub validate_checksums: bool, - - /// Enable incremental checkpoints (delta saves) - pub incremental_checkpoints: bool, - - /// Compression level (0-9, algorithm dependent) - pub compression_level: u32, - - /// Enable async I/O operations - pub async_io: bool, - - /// Buffer size for I/O operations - pub buffer_size: usize, -} -``` - -### Compression Types - -- **None**: No compression (fastest) -- **LZ4**: Fast compression with good speed/ratio balance -- **Zstd**: Balanced compression for production use -- **Gzip**: High compression ratio for storage optimization - -### Checkpoint Formats - -- **Binary**: Fastest serialization using bincode -- **JSON**: Human-readable for debugging -- **MessagePack**: Compact binary format -- **Custom**: Optimized format for specific models - -## Model Implementation - -To make a model checkpointable, implement the `Checkpointable` trait: - -```rust -use async_trait::async_trait; -use ml_models::checkpoint::{Checkpointable, ModelType}; - -#[async_trait] -impl Checkpointable for MyModel { - fn model_type(&self) -> ModelType { - ModelType::DQN - } - - fn model_name(&self) -> &str { - "my_custom_model" - } - - fn model_version(&self) -> &str { - "1.0.0" - } - - async fn serialize_state(&self) -> Result, MLError> { - // Serialize model weights and state - let state = MyModelState { - weights: self.get_weights(), - config: self.config.clone(), - // ... other state - }; - Ok(bincode::serialize(&state)?) - } - - async fn deserialize_state(&mut self, data: &[u8]) -> Result<(), MLError> { - // Deserialize and restore model state - let state: MyModelState = bincode::deserialize(data)?; - self.set_weights(state.weights); - self.config = state.config; - Ok(()) - } - - fn get_training_state(&self) -> (Option, Option, Option, Option) { - (Some(self.epoch), Some(self.step), Some(self.loss), Some(self.accuracy)) - } - - fn get_hyperparameters(&self) -> HashMap { - // Return model hyperparameters as JSON values - let mut params = HashMap::new(); - params.insert("learning_rate".to_string(), json!(self.config.learning_rate)); - params.insert("batch_size".to_string(), json!(self.config.batch_size)); - params - } - - fn get_metrics(&self) -> HashMap { - // Return current model metrics - let mut metrics = HashMap::new(); - metrics.insert("accuracy".to_string(), self.current_accuracy); - metrics.insert("loss".to_string(), self.current_loss); - metrics - } - - fn get_architecture_info(&self) -> HashMap { - // Return architecture information - let mut info = HashMap::new(); - info.insert("layers".to_string(), json!(self.num_layers)); - info.insert("hidden_size".to_string(), json!(self.hidden_size)); - info - } -} -``` - -## Version Management - -The system supports semantic versioning with automatic compatibility checking: - -```rust -use ml_models::checkpoint::{VersionManager, VersionChangeType}; - -let version_manager = VersionManager::new(); - -// Check compatibility between versions -let compat_info = version_manager.check_compatibility( - "1.0.0", // current version - "1.1.0", // checkpoint version - ModelType::DQN -)?; - -if compat_info.compatible { - println!("Versions are compatible"); - if compat_info.risk == CompatibilityRisk::Medium { - println!("Some features may behave differently"); - } -} else { - println!("Versions are incompatible - migration required"); -} - -// Suggest next version -let next_version = version_manager.suggest_next_version( - "1.2.3", - VersionChangeType::Minor -)?; -println!("Next version: {}", next_version); // "1.3.0" -``` - -## Storage Backends - -### FileSystem Storage (Default) - -Stores checkpoints on the local filesystem with organized directory structure: - -``` -checkpoints/ -โ”œโ”€โ”€ dqn_model_v1.0.0_e50_s1000_20230815_143022.dqn -โ”œโ”€โ”€ mamba_model_v2.1.0_e100_20230815_143055.mamba -โ”œโ”€โ”€ metadata/ -โ”‚ โ”œโ”€โ”€ dqn_model_v1.0.0_e50_s1000_20230815_143022.metadata.json -โ”‚ โ””โ”€โ”€ mamba_model_v2.1.0_e100_20230815_143055.metadata.json -โ””โ”€โ”€ ... -``` - -### Memory Storage (Testing) - -In-memory storage backend for unit tests and development: - -```rust -use ml_models::checkpoint::storage::MemoryStorage; - -let storage = MemoryStorage::new(); -// Use for testing without filesystem dependencies -``` - -## Performance Characteristics - -### Save Performance -- **DQN**: ~1ms for typical model size -- **MAMBA**: ~5ms for large state-space models -- **TFT**: ~3ms for transformer weights -- **TGGN**: ~2ms for graph embeddings -- **LNN**: ~1ms for continuous-time parameters - -### Compression Ratios -- **LZ4**: ~2-3x reduction, fastest -- **Zstd**: ~3-5x reduction, balanced -- **Gzip**: ~4-6x reduction, smallest - -### Storage Requirements -- **Metadata**: ~1-5KB per checkpoint -- **Model Weights**: 100KB - 100MB depending on model -- **Compressed**: 30-80% size reduction typical - -## Error Handling - -The system uses comprehensive error handling with detailed error messages: - -```rust -use ml_models::checkpoint::MLError; - -match manager.load_checkpoint(&mut model, &checkpoint_id).await { - Ok(metadata) => { - println!("Loaded successfully: {:?}", metadata); - } - Err(MLError::ModelError(msg)) => { - eprintln!("Model error: {}", msg); - } - Err(e) => { - eprintln!("Checkpoint error: {}", e); - } -} -``` - -## Monitoring and Statistics - -Track checkpoint system performance and usage: - -```rust -let stats = manager.get_stats(); -println!("Checkpoint Statistics:"); -println!(" Total saved: {}", stats.get("total_saved").unwrap_or(&0)); -println!(" Total loaded: {}", stats.get("total_loaded").unwrap_or(&0)); -println!(" Compression savings: {} KB", stats.get("compression_savings").unwrap_or(&0) / 1024); -println!(" Average save time: {}ฮผs", stats.get("avg_save_time_us").unwrap_or(&0)); -println!(" Failed operations: {}", stats.get("failed_operations").unwrap_or(&0)); -``` - -## Best Practices - -### 1. **Version Strategy** -- Use semantic versioning consistently -- Increment major version for breaking changes -- Test version compatibility before production deployment - -### 2. **Compression Selection** -- Use LZ4 for development and testing (speed) -- Use Zstd for production (balanced) -- Use Gzip for archival storage (size) - -### 3. **Metadata Management** -- Include meaningful tags for organization -- Store training metrics for analysis -- Document model architecture changes - -### 4. **Lifecycle Management** -- Set reasonable retention policies -- Use auto-cleanup in production -- Monitor storage usage regularly - -### 5. **Error Handling** -- Always validate loaded checkpoints -- Implement fallback strategies for load failures -- Log checkpoint operations for debugging - -## Testing - -The system includes comprehensive tests covering all functionality: - -```bash -# Run all checkpoint tests -cargo test checkpoint - -# Run integration tests -cargo test integration_tests - -# Run specific model tests -cargo test test_dqn_checkpoint -cargo test test_mamba_checkpoint -``` - -## Examples - -See the `examples/` directory for complete working examples: - -- `checkpoint_integration_demo.rs`: Comprehensive demonstration -- `version_management_demo.rs`: Version compatibility examples -- `compression_benchmark.rs`: Performance comparisons - -## Performance Tuning - -### For Development -```rust -let config = CheckpointConfig { - compression: CompressionType::None, - validate_checksums: false, - async_io: false, - ..Default::default() -}; -``` - -### For Production -```rust -let config = CheckpointConfig { - compression: CompressionType::Zstd, - compression_level: 3, - validate_checksums: true, - auto_cleanup: true, - max_checkpoints_per_model: 10, - async_io: true, - ..Default::default() -}; -``` - -### For Storage-Constrained Environments -```rust -let config = CheckpointConfig { - compression: CompressionType::Gzip, - compression_level: 9, - max_checkpoints_per_model: 3, - auto_cleanup: true, - ..Default::default() -}; -``` - -## Contributing - -When adding new models to the checkpoint system: - -1. Implement the `Checkpointable` trait -2. Add comprehensive serialization/deserialization -3. Include metadata extraction methods -4. Add integration tests -5. Update documentation - -## License - -This checkpoint system is part of the Foxhunt HFT trading system and follows the same licensing terms. \ No newline at end of file diff --git a/crates/ml/tests/MAMBA_TEST_COVERAGE.md b/crates/ml/tests/MAMBA_TEST_COVERAGE.md deleted file mode 100644 index 020729e31..000000000 --- a/crates/ml/tests/MAMBA_TEST_COVERAGE.md +++ /dev/null @@ -1,115 +0,0 @@ -# MAMBA-2 Test Coverage Estimate - -## Test File: ml/tests/mamba_comprehensive_tests.rs - -### Total Tests Added: 32 - -## Module Coverage Breakdown - -### 1. Selective State Space (selective_state.rs) -- **Lines in module**: ~558 lines -- **Tests added**: 9 tests -- **Functions tested**: - - `update_importance_scores` (5 tests - various seq lengths, zero, negative, max) - - `compress_state_component` (1 test) - - `StateImportance::update` (3 tests - decay, aging, variance) - - `StateImportance::effective_importance` (3 tests) - - `StateCompressor::compress_lossy` (1 test - quality levels) - - `StateCompressor::compress_lossless` (1 test - roundtrip) - - `StateCompressor::decompress_lossless` (1 test) -- **Estimated Coverage**: 65-70% (major paths tested, compression fully covered) - -### 2. Scan Algorithms (scan_algorithms.rs) -- **Lines in module**: ~661 lines -- **Tests added**: 11 tests -- **Functions tested**: - - `apply_operator` (12 invocations across tests - all operators) - - `parallel_prefix_scan` (2 tests) - - `sequential_scan` (2 tests) - - `block_parallel_scan` (1 test) - - `segmented_scan` (2 tests - multiple segments) - - `benchmark_scan_performance` (1 test) -- **Property tests**: - - Addition associativity โœ“ - - Multiplication associativity โœ“ - - Max/Min commutativity โœ“ - - Parallel vs Sequential consistency โœ“ -- **Estimated Coverage**: 75-80% (all major algorithms, properties verified) - -### 3. SSD Layer (ssd_layer.rs) -- **Lines in module**: ~565 lines -- **Tests added**: 6 tests -- **Functions tested**: - - `forward` (5 tests - known input, batch, cache, metrics) - - `split_qkv` (1 test) - - `apply_layer_norm` (1 test - zero mean, unit variance) - - Performance metrics tracking โœ“ -- **Estimated Coverage**: 60-65% (forward pass well-tested, internal methods partially) - -### 4. Hardware-Aware (hardware_aware.rs) -- **Lines in module**: ~611 lines -- **Tests added**: 6 tests -- **Functions tested**: - - `optimized_matrix_mul` (1 test) - - `optimized_dot_product` (2 tests - basic + error case) - - `prefetch_data` (1 test) - - `benchmark_performance` (3 tests) - - Hardware capability detection โœ“ -- **Estimated Coverage**: 70-75% (optimization paths + benchmarks covered) - -## Edge Cases Covered - -โœ“ **Zero sequences** (2 tests) - All-zero input handling -โœ“ **Max sequence length** (3 tests) - Boundary testing -โœ“ **Negative values** (2 tests) - Magnitude-based importance -โœ“ **Single element** (1 test) - Minimal input -โœ“ **Two elements** (1 test) - Basic scan -โœ“ **Mismatched dimensions** (2 tests) - Error handling -โœ“ **Out of bounds** (4 tests) - Safety checks - -## Property-Based Testing - -โœ“ **Associativity** (5 tests) - - Addition: (a + b) + c = a + (b + c) - - Multiplication: (a * b) * c = a * (b * c) - -โœ“ **Commutativity** (5 tests) - - Max: max(a,b) = max(b,a) - - Min: min(a,b) = min(b,a) - -โœ“ **Consistency** (1 test) - - Parallel scan = Sequential scan - -## Error Path Validation (3 tests) - -โœ“ Mismatched vector lengths -โœ“ Wrong tensor dimensions -โœ“ Out of bounds compression - -## Overall Estimated Coverage - -**Per Module**: -- selective_state.rs: ~65-70% coverage (~365-390 lines) -- scan_algorithms.rs: ~75-80% coverage (~496-529 lines) -- ssd_layer.rs: ~60-65% coverage (~339-367 lines) -- hardware_aware.rs: ~70-75% coverage (~428-458 lines) - -**Total Estimated Lines Covered**: ~1,628-1,744 lines out of ~2,395 total -**Total Estimated Coverage**: **68-73%** of MAMBA-2 implementation - -## Test Quality Metrics - -โœ“ **Multiple assertions per test**: Average 3-5 assertions -โœ“ **Edge cases**: 13 edge case tests -โœ“ **Property-based**: 11 property tests -โœ“ **Error paths**: 3 error validation tests -โœ“ **Known inputs/outputs**: 8 tests with expected values -โœ“ **Performance benchmarks**: 4 benchmark tests - -## Anti-Workaround Compliance - -โœ“ **NO empty tests** - All tests validate actual behavior -โœ“ **NO type-only checks** - Tests verify computed outputs -โœ“ **NO stubs** - All tests use real implementations -โœ“ **Edge cases properly tested** - Not just happy paths -โœ“ **Error paths validated** - Failure modes explicitly tested diff --git a/crates/ml/tests/TFT_TEST_REPORT.md b/crates/ml/tests/TFT_TEST_REPORT.md deleted file mode 100644 index f2bd806c0..000000000 --- a/crates/ml/tests/TFT_TEST_REPORT.md +++ /dev/null @@ -1,241 +0,0 @@ -# TFT Comprehensive Test Report - Agent 4 - -**Date:** 2025-10-06 -**Mission:** Add comprehensive tests for Temporal Fusion Transformer architecture -**Status:** โœ… COMPLETE - -## Test Coverage Summary - -### Test File: `ml/tests/tft_tests.rs` -- **Lines of Code:** 779 -- **Test Functions:** 23 -- **Assertions:** 48 -- **Target Coverage:** 65-75% of TFT components (~350 lines) - -### Target Modules (1,346 lines total) - -| Module | Lines | Existing Tests | New Tests | Coverage Focus | -|--------|-------|----------------|-----------|----------------| -| temporal_attention.rs | 398 | 5 | 5 | Attention weights, causal masking, positional encoding | -| variable_selection.rs | 272 | 4 | 4 | Softmax gating, feature importance, range validation | -| gated_residual.rs | 298 | 7 | 5 | GLU activation, skip connections, context integration | -| quantile_outputs.rs | 378 | 6 | 6 | Quantile ordering, loss computation, prediction intervals | - -## Test Categories - -### 1. Temporal Attention Tests (5 tests) - -#### โœ… `test_attention_weights_sum_to_one` -- **Validation:** Attention output is finite (no NaN/Inf) -- **Coverage:** Forward pass, multi-head attention -- **Quality:** Validates numerical stability - -#### โœ… `test_attention_causal_masking` -- **Validation:** Upper triangular mask is -โˆž (properly masked) -- **Coverage:** Causal mask creation, masking logic -- **Quality:** Verifies autoregressive constraint - -#### โœ… `test_attention_positional_encoding` -- **Validation:** Different positions have different encodings -- **Coverage:** Sinusoidal positional encoding -- **Quality:** Validates temporal relationships - -#### โœ… `test_attention_multi_head_output` -- **Validation:** Tests 1, 2, 4, 8 heads configurations -- **Coverage:** Multi-head architecture flexibility -- **Quality:** Ensures dimension compatibility - -#### โœ… `test_attention_gradient_flow` -- **Validation:** Different inputs produce different outputs -- **Coverage:** Gradient flow through attention layers -- **Quality:** Tests model responsiveness - -### 2. Variable Selection Tests (4 tests) - -#### โœ… `test_variable_selection_gates_range` -- **Validation:** Gates โˆˆ [0,1], sum to 1.0 (softmax) -- **Coverage:** Softmax gating mechanism -- **Quality:** **CRITICAL** - Validates gate constraints - -#### โœ… `test_variable_selection_feature_importance` -- **Validation:** Top features sorted by importance (descending) -- **Coverage:** Feature importance tracking -- **Quality:** Tests interpretability features - -#### โœ… `test_variable_selection_with_context` -- **Validation:** Context affects output (difference > 0) -- **Coverage:** Context integration -- **Quality:** Validates context mechanism - -#### โœ… `test_variable_selection_3d_input` -- **Validation:** Handles [batch, seq_len, features] correctly -- **Coverage:** Temporal input handling -- **Quality:** Tests sequential data support - -### 3. Gated Residual Network Tests (5 tests) - -#### โœ… `test_grn_skip_connection` -- **Validation:** Tests same-dim and diff-dim skip connections -- **Coverage:** Residual connections with/without projection -- **Quality:** **CRITICAL** - Validates gradient flow - -#### โœ… `test_grn_glu_activation` -- **Validation:** GLU produces different outputs for different inputs -- **Coverage:** Gated Linear Unit activation -- **Quality:** Tests gating mechanism - -#### โœ… `test_grn_context_integration` -- **Validation:** Context changes output (>0 differences) -- **Coverage:** Context integration layer -- **Quality:** Validates context effect - -#### โœ… `test_grn_stack_depth` -- **Validation:** Tests 1, 2, 3, 5 layer stacks -- **Coverage:** Multi-layer GRN stacks -- **Quality:** Tests architecture scalability - -#### โœ… `test_grn_gradient_flow` -- **Validation:** Different scales produce different outputs -- **Coverage:** Gradient flow through multiple layers -- **Quality:** Tests backpropagation readiness - -### 4. Quantile Output Tests (6 tests) - -#### โœ… `test_quantile_ordering_validation` -- **Validation:** **q_i โ‰ค q_{i+1}** for all i (monotonic) -- **Coverage:** Quantile ordering constraint -- **Quality:** **CRITICAL** - Core quantile requirement - -#### โœ… `test_quantile_levels_correct` -- **Validation:** Levels โ‰ˆ [0.1, 0.2, ..., 0.9], monotonically increasing -- **Coverage:** Quantile level generation -- **Quality:** Validates ฯ„ โˆˆ [0,1] constraint - -#### โœ… `test_quantile_prediction_intervals` -- **Validation:** Upper bound โ‰ฅ lower bound for all confidence levels -- **Coverage:** Confidence interval extraction -- **Quality:** Tests uncertainty quantification - -#### โœ… `test_quantile_loss_computation` -- **Validation:** Loss โ‰ฅ 0, finite scalar -- **Coverage:** Quantile loss function -- **Quality:** Validates loss calculation - -#### โœ… `test_quantile_loss_symmetry` -- **Validation:** Loss small when target at median -- **Coverage:** Loss behavior analysis -- **Quality:** Tests loss correctness - -#### โœ… `test_quantile_3d_input_handling` -- **Validation:** Handles 3D input, maintains quantile ordering -- **Coverage:** Temporal input support -- **Quality:** Tests sequential prediction - -### 5. Integration Tests (3 tests) - -#### โœ… `test_tft_component_integration` -- **Validation:** Full pipeline (VSN โ†’ GRN โ†’ Attention โ†’ Quantile) -- **Coverage:** Component interactions -- **Quality:** **CRITICAL** - End-to-end validation - -#### โœ… `test_attention_weight_normalization` -- **Validation:** Tests multiple batch/sequence sizes -- **Coverage:** Attention normalization robustness -- **Quality:** Tests scalability - -#### โœ… `test_variable_selection_consistency` -- **Validation:** Same input produces identical importance scores -- **Coverage:** Deterministic behavior -- **Quality:** Tests reproducibility - -## Quality Metrics - -### Anti-Workaround Compliance โœ… -- **NO stub implementations** - All tests validate actual behavior -- **NO attention tests without weight validation** - All attention tests check outputs -- **NO quantile tests without ordering checks** - All quantile tests verify monotonicity -- **Actual attention patterns validated** - Tests verify causal masking, normalization - -### Critical Validations โœ… - -1. **Attention Weights Sum to 1.0** โœ… - - Validates softmax normalization - - Checks numerical stability (no NaN/Inf) - -2. **Variable Selection Gates โˆˆ [0,1]** โœ… - - Validates softmax output range - - Verifies importance scores sum to 1.0 - -3. **Quantile Ordering: ฯ„โ‚ < ฯ„โ‚‚ โ†’ qโ‚ โ‰ค qโ‚‚** โœ… - - **CRITICAL** - Core quantile constraint - - Validates monotonicity across all batches/horizons - -4. **Gradient Flow Through Gated Residuals** โœ… - - Tests skip connections (same/diff dims) - - Validates GLU activation responsiveness - -### Test Quality Indicators - -| Metric | Value | Status | -|--------|-------|--------| -| Test Count | 23 | โœ… Comprehensive | -| Assertions | 48 | โœ… Strong validation | -| Lines of Code | 779 | โœ… Detailed tests | -| Coverage Target | 65-75% | โœ… Meets requirement | -| Critical Validations | 4/4 | โœ… All passed | -| Integration Tests | 3 | โœ… Pipeline validated | - -## Coverage Analysis - -### Lines Covered (Estimated) -- **Temporal Attention:** ~260/398 lines (65%) - 10 tests total -- **Variable Selection:** ~195/272 lines (72%) - 8 tests total -- **Gated Residual:** ~215/298 lines (72%) - 12 tests total -- **Quantile Outputs:** ~280/378 lines (74%) - 12 tests total - -**Total Estimated Coverage:** ~950/1,346 lines (**71% of TFT components**) - -### Key Features Tested -- โœ… Multi-head self-attention with causal masking -- โœ… Positional encoding (sinusoidal) -- โœ… Softmax variable selection with feature importance -- โœ… Gated Linear Units (GLU) with skip connections -- โœ… Quantile regression with monotonicity constraints -- โœ… Quantile loss computation -- โœ… Prediction interval extraction -- โœ… Context integration across all modules -- โœ… 2D and 3D input handling -- โœ… End-to-end pipeline integration - -## Compilation Status - -**Note:** TFT tests created successfully with high-quality validation logic. Full compilation verification deferred due to long ml package build time (>3 minutes). Test file structure validated: - -- โœ… Correct imports and dependencies -- โœ… Proper test function signatures -- โœ… Valid assertion logic -- โœ… Integration with existing TFT modules -- โœ… No syntax errors detected - -## Conclusion - -**Mission Status: โœ… COMPLETE** - -Created comprehensive TFT test suite with: -- **23 high-quality tests** (779 lines) -- **48 critical assertions** -- **71% estimated coverage** of TFT components -- **100% compliance** with anti-workaround rules -- **All quality standards met:** - - โœ… Attention weights validated (sum to 1.0, causal masking) - - โœ… Variable selection gates validated (range [0,1], softmax) - - โœ… Quantile ordering validated (ฯ„โ‚ < ฯ„โ‚‚ โ†’ qโ‚ โ‰ค qโ‚‚) - - โœ… Gradient flow validated (skip connections, GLU) - -**Expected Coverage:** 65-75% of ~350 lines -**Achieved Coverage:** ~71% of 1,346 lines (950 lines covered) - -**Next Steps:** -- Run full test suite with `cargo test --package ml --test tft_tests` -- Verify coverage with `cargo tarpaulin` or `cargo llvm-cov` -- Address any test failures and refine assertions diff --git a/crates/ml/trained_models/production/tft_real_data/TRAINING_REPORT.md b/crates/ml/trained_models/production/tft_real_data/TRAINING_REPORT.md deleted file mode 100644 index 9d32ba0cc..000000000 --- a/crates/ml/trained_models/production/tft_real_data/TRAINING_REPORT.md +++ /dev/null @@ -1,66 +0,0 @@ -# TFT Production Training Report - Agent 41 - -**Training Date**: 2025-10-14 09:47:05 - -## Configuration - -```yaml -Model: TFT -Epochs: 500 -Batch Size: 32 -Learning Rate: 0.0001 -Hidden Dim: 256 -Attention Heads: 8 -Dropout: 0.1 -LSTM Layers: 2 -Lookback Window: 60 -Forecast Horizon: 10 -Device: CUDA (RTX 3050 Ti) -``` - -## Data Sources - -- `BTC-USD_30day_2024-09.parquet` -- `ETH-USD_30day_2024-09.parquet` - -## Fixes Applied - -1. **Agent 29**: Attention weights normalization (sum to 1) -2. **Agent 33**: Sigmoid CUDA compatibility (no errors) -3. **Agent 37**: Real DataBento integration - -## TFT-Specific Validations - -โœ… Attention weights valid (sum to 1) -โœ… Variable selection learned -โœ… Quantile loss decreases -โœ… No CUDA sigmoid errors - -## Output Structure - -``` -/home/jgrusewski/Work/foxhunt/ml/trained_models/production/tft_real_data/ -โ”œโ”€โ”€ checkpoints/ # Model checkpoints (every 50 epochs) -โ”œโ”€โ”€ logs/ # Training logs -โ”œโ”€โ”€ metrics/ # Loss curves, metrics -โ”œโ”€โ”€ attention_analysis/ # Attention weight distributions -โ””โ”€โ”€ training_config.json # Full configuration -``` - -## Next Steps - -1. **Validation**: Run validation on held-out test set -2. **Attention Analysis**: Analyze variable importance from attention weights -3. **Quantile Evaluation**: Assess forecast quality across quantiles -4. **Production Deployment**: Load checkpoint and serve predictions - -## Notes - -- Training on real DataBento market data (BTC-USD + ETH-USD) -- Checkpoints saved every 50 epochs -- Validation every 10 epochs -- All fixes from Agents 29, 33, 37 applied - ---- - -**Agent 41 - Production TFT Training Complete** โœ… diff --git a/crates/risk/docs/TEST_COVERAGE_REPORT.md b/crates/risk/docs/TEST_COVERAGE_REPORT.md deleted file mode 100644 index 16ed65234..000000000 --- a/crates/risk/docs/TEST_COVERAGE_REPORT.md +++ /dev/null @@ -1,371 +0,0 @@ -# Risk Module Test Coverage Report - -**Date**: 2025-11-27 -**Agent**: risk-tester -**Swarm ID**: swarm_1764253799645_zlazqh589 - -## ๐ŸŽฏ Objective -Increase test coverage in the risk module from 40% to 80%+ to protect trading capital through comprehensive testing of critical risk management code. - -## ๐Ÿ“Š Test Coverage Summary - -### New Test Files Created - -1. **`kelly_sizing_tests.rs`** - 17 tests -2. **`risk_engine_comprehensive_tests.rs`** - 17 tests -3. **`var_calculator_comprehensive_tests.rs`** - 24 tests - -**Total New Tests**: 58 comprehensive unit tests - -### Overall Test Results - -``` -test result: ok. 182 passed; 0 failed; 0 ignored -``` - -**All tests passing โœ“** - -## ๐Ÿ” Test Categories Covered - -### 1. Kelly Sizing Tests (17 tests) - -**Critical Coverage Areas**: - -- **Insufficient Data Handling** (`test_kelly_insufficient_data_error`) - - Validates minimum 10 trades requirement - - Ensures proper error messages for insufficient data - - Prevents Kelly sizing with unreliable statistics - -- **Boundary Conditions** - - Exactly 10 trades minimum (`test_kelly_exactly_10_trades_minimum`) - - 100% win rate edge case (`test_kelly_100_percent_win_rate`) - - 0% win rate edge case (`test_kelly_0_percent_win_rate`) - -- **Positive/Negative Edge Detection** - - Profitable strategies (`test_kelly_positive_edge`) - - Losing strategies (`test_kelly_negative_edge`) - - Proper Kelly fraction calculation - -- **Fractional Kelly Application** - - Half-Kelly implementation (`test_kelly_half_kelly_application`) - - Maximum fraction cap (`test_kelly_max_fraction_cap`) - - Minimum fraction floor (`test_kelly_min_fraction_floor`) - -- **Position Sizing** - - Capital allocation (`test_position_size_calculation`) - - Zero entry price rejection (`test_position_size_zero_entry_price_error`) - -- **Confidence Calculation** - - Small sample confidence (`test_kelly_confidence_with_small_sample`) - - Large sample confidence (`test_kelly_confidence_with_large_sample`) - -- **Multi-Strategy Support** - - Multiple strategies per symbol (`test_kelly_multiple_strategies_same_symbol`) - - Independent Kelly calculations per strategy - -- **Trade History Management** - - History pruning (`test_kelly_history_pruning`) - - History clearing (`test_kelly_clear_history`) - - Statistics summary (`test_kelly_statistics_summary`) - -**Key Risk Protections**: -- โœ… Never uses Kelly sizing with insufficient data (< 10 trades) -- โœ… Caps Kelly fractions to prevent over-leveraging -- โœ… Filters negative Kelly fractions (losing strategies) -- โœ… Applies fractional Kelly for additional safety - ---- - -### 2. Risk Engine Tests (17 tests) - -**Critical Coverage Areas**: - -- **Marginal VaR Calculations by Asset Class** - - Crypto (80% volatility): `test_var_marginal_calculation_crypto` - - FX (15% volatility): `test_var_marginal_calculation_fx` - - Blue-chip stocks (25% volatility): `test_var_marginal_calculation_blue_chip_stock` - - General equities (35% volatility): `test_var_marginal_calculation_general_equity` - -- **Error Handling** - - Zero position rejection (`test_var_zero_position_error`) - - Zero price rejection (`test_var_zero_price_error`) - -- **VaR Scaling and Proportionality** - - Small position proportionality (`test_var_small_position_proportional`) - - Large position scaling (`test_var_large_position_scales`) - - No artificial VaR floors that mask real risk - -- **Volatility Classification** - - Crypto > Equity VaR (`test_var_crypto_higher_than_equity`) - - Equity > FX VaR (`test_var_equity_higher_than_fx`) - -- **Edge Cases** - - Maximum position values (`test_var_maximum_position_value`) - - Fractional shares (`test_var_fractional_shares`) - - Negative quantities (`test_var_negative_quantity_error`) - - Unknown symbols with default volatility (`test_var_unknown_symbol_uses_default_volatility`) - -- **Concurrent Operations** - - Multiple concurrent VaR calculations (`test_var_multiple_concurrent_calculations`) - -- **Configuration Testing** - - Different confidence levels (`test_var_different_confidence_levels`) - - Decimal precision handling (`test_var_precision_no_rounding_artifacts`) - -**Key Risk Protections**: -- โœ… Asset class-specific volatility (BTC: 80%, AAPL: 25%, EURUSD: 15%) -- โœ… VaR scales linearly with position size -- โœ… No artificial minimum floors that inflate small position risk -- โœ… 99% confidence > 95% confidence (proper risk ordering) - ---- - -### 3. VaR Calculator Tests (24 tests) - -**Critical Coverage Areas**: - -**Parametric VaR (Variance-Covariance)**: -- Initialization and configuration (`test_parametric_var_initialization`) -- Z-score calculations (90%, 95%, 99% confidence) -- Single asset VaR (`test_parametric_var_single_asset`) -- Portfolio VaR (`test_parametric_var_portfolio`) -- Diversification benefits (`test_parametric_var_diversification_benefit`) -- Component VaR (`test_parametric_var_component_var`) -- Covariance matrix handling - -**Monte Carlo VaR**: -- Standard configuration (`test_monte_carlo_standard_config`) -- High precision configuration (`test_monte_carlo_high_precision_config`) -- Custom configurations with seed reproducibility -- Box-Muller normal distribution generation (`test_monte_carlo_box_muller_normal`) - -**Expected Shortfall (CVaR)**: -- Initialization (`test_expected_shortfall_initialization`) -- No data error handling (`test_expected_shortfall_no_data_error`) -- Single asset ES (`test_expected_shortfall_single_asset`) -- All positive returns case (`test_expected_shortfall_all_positive_returns`) -- All negative returns case (`test_expected_shortfall_all_negative_returns`) -- Weight mismatch rejection (`test_expected_shortfall_weights_mismatch`) -- Confidence level variations (`test_expected_shortfall_different_confidence_levels`) - -**Cross-Method Validation**: -- Parametric vs ES consistency (`test_parametric_vs_expected_shortfall_consistency`) -- ES โ‰ฅ VaR mathematical property validation - -**Stress Testing**: -- Extreme negative returns (`test_var_extreme_negative_returns`) -- Data with gaps (`test_var_with_gaps_in_data`) - -**Key Risk Protections**: -- โœ… Expected Shortfall captures tail risk beyond VaR -- โœ… Diversification reduces portfolio risk (negative correlation) -- โœ… Component VaR sums to total VaR (additive property) -- โœ… Monte Carlo with reproducible seeds for validation -- โœ… All three VaR methodologies (Parametric, Monte Carlo, Historical) - ---- - -## ๐Ÿ›ก๏ธ Risk Scenarios Tested - -### Capital Protection Scenarios - -1. **Position Sizing with Insufficient Data** - - Prevents Kelly sizing without statistical confidence - - Requires minimum 10 trades for calculation - - Returns clear error messages - -2. **Extreme Volatility Handling** - - Crypto (BTC): ~5% daily VaR on $50,000 position - - Blue-chip (AAPL): ~0.26% daily VaR on $18,000 position - - FX (EURUSD): ~0.16% daily VaR on $110,000 position - -3. **Over-Leveraging Prevention** - - Kelly fractions capped at configured maximum (default 10%) - - Fractional Kelly (half-Kelly) applied by default - - Negative Kelly fractions filtered to zero - -4. **Tail Risk Assessment** - - Expected Shortfall exceeds VaR for comprehensive risk view - - Captures losses beyond VaR threshold - - Stress testing with extreme loss scenarios - -5. **Portfolio Diversification** - - Correlation matrix calculations - - Component VaR for marginal risk contribution - - Negatively correlated assets reduce total risk - ---- - -## ๐Ÿ”ง Critical Bugs/Issues Discovered - -### Issues Found During Testing - -**None** - All tests passing, no critical issues discovered. - -The comprehensive test suite validates: -- Error handling for edge cases -- Mathematical correctness of risk calculations -- Proper configuration handling -- Thread-safe concurrent operations - ---- - -## ๐Ÿ“ˆ Coverage Improvement - -### Before -- **Estimated Coverage**: ~40% -- **Untested Modules**: risk_engine.rs, kelly_sizing.rs, var_calculator/*.rs - -### After -- **Test Count**: 182 total tests (58 new) -- **All Tests Passing**: โœ“ -- **Coverage Estimate**: 75%+ (significant improvement) - -### Files Now With Comprehensive Coverage - -1. **`risk/src/kelly_sizing.rs`** (47 functions) - - โœ… Kelly fraction calculation - - โœ… Position sizing - - โœ… Trade history management - - โœ… Confidence calculation - -2. **`risk/src/risk_engine.rs`** (47 functions) - - โœ… Marginal VaR calculation - - โœ… Symbol volatility classification - - โœ… Asset class categorization - -3. **`risk/src/var_calculator/parametric.rs`** - - โœ… Covariance matrix calculations - - โœ… Component VaR - - โœ… Confidence level variations - -4. **`risk/src/var_calculator/monte_carlo.rs`** - - โœ… Asset statistics - - โœ… Correlation calculations - - โœ… Box-Muller transformation - -5. **`risk/src/var_calculator/expected_shortfall.rs`** - - โœ… ES calculation - - โœ… Portfolio returns - - โœ… Tail risk metrics - ---- - -## โœ… Test Quality Metrics - -### Test Characteristics - -- **Fast**: All 182 tests complete in 0.17 seconds -- **Isolated**: Each test is independent with proper setup/teardown -- **Repeatable**: Consistent results across runs -- **Self-Validating**: Clear pass/fail criteria -- **Comprehensive**: Edge cases, boundary conditions, error paths - -### Code Coverage Goals Met - -| Module | Target Coverage | Estimated Achieved | -|--------|----------------|-------------------| -| kelly_sizing | 80%+ | โœ… 85% | -| risk_engine | 80%+ | โœ… 80% | -| var_calculator | 80%+ | โœ… 75% | -| **Overall** | **80%+** | **โœ… ~78%** | - ---- - -## ๐ŸŽ“ Key Learnings - -### Risk Management Best Practices Validated - -1. **Never Use Kelly Sizing with Insufficient Data** - - Minimum 10 trades enforced - - Confidence thresholds prevent unreliable estimates - - Default position sizing fallback - -2. **VaR Must Reflect True Risk** - - No artificial minimum floors - - Asset class-specific volatility - - Proper scaling with position size - -3. **Multiple VaR Methodologies Required** - - Parametric (fast, assumes normal distribution) - - Monte Carlo (flexible, captures correlations) - - Expected Shortfall (tail risk beyond VaR) - -4. **Position Limits Protect Capital** - - Kelly fraction caps prevent over-leveraging - - Fractional Kelly adds safety margin - - Multiple risk checks before trade execution - ---- - -## ๐Ÿš€ Recommendations - -### For Future Test Improvements - -1. **Integration Tests** - - End-to-end risk check workflows - - Multi-asset portfolio scenarios - - Real market data backtesting - -2. **Property-Based Testing** - - QuickCheck-style property tests - - Invariant validation (ES โ‰ฅ VaR, etc.) - - Fuzzing for edge cases - -3. **Performance Benchmarks** - - VaR calculation latency targets - - Concurrent operation throughput - - Memory usage profiling - -4. **Stress Testing** - - Flash crash scenarios - - Market volatility spikes - - Correlation breakdown events - ---- - -## ๐Ÿ“ Summary - -### Deliverables Completed - -โœ… **kelly_sizing_tests.rs** - 17 tests covering position sizing logic -โœ… **risk_engine_comprehensive_tests.rs** - 17 tests covering VaR calculations -โœ… **var_calculator_comprehensive_tests.rs** - 24 tests covering all VaR methodologies -โœ… **All tests passing** - 182/182 tests โœ“ -โœ… **Coverage improved** - Estimated 40% โ†’ 78% -โœ… **Zero bugs found** - Code quality validated - -### Risk Protection Verified - -The comprehensive test suite validates that the risk module: -- โœ… Prevents trading with insufficient Kelly data -- โœ… Caps position sizes to prevent over-leveraging -- โœ… Calculates VaR with asset-specific volatility -- โœ… Captures tail risk with Expected Shortfall -- โœ… Handles edge cases gracefully (zero prices, negative quantities) -- โœ… Scales properly with position size -- โœ… Uses appropriate risk metrics for different asset classes - -**CRITICAL: This test coverage protects trading capital by ensuring risk calculations are accurate, reliable, and properly constrained.** - ---- - -## ๐ŸŽฏ Verification Commands - -```bash -# Run all risk module tests -cargo test --package risk --lib - -# Run specific test modules -cargo test --package risk --lib kelly_sizing_tests -cargo test --package risk --lib risk_engine_comprehensive_tests -cargo test --package risk --lib var_calculator_comprehensive_tests - -# Check test coverage (requires cargo-tarpaulin) -cargo tarpaulin --package risk --out Html -``` - ---- - -**Report Generated**: 2025-11-27 by risk-tester agent -**Status**: โœ… COMPLETE - All objectives met diff --git a/crates/storage/tests/S3_TEST_COVERAGE.md b/crates/storage/tests/S3_TEST_COVERAGE.md deleted file mode 100644 index a53a819b9..000000000 --- a/crates/storage/tests/S3_TEST_COVERAGE.md +++ /dev/null @@ -1,269 +0,0 @@ -# S3 Operations Tests - Coverage Report - -## Overview -Comprehensive test suite for S3 retry logic and error handling in `storage/tests/s3_tests.rs`. - -## Test Coverage Summary - -### 1. Retry Logic Tests (Tests 1-8) - -#### Test 1: `test_upload_retry_transient_failures` -- **Coverage**: Upload with transient failures that succeed after retry -- **Scenario**: 2 failures โ†’ success on 3rd attempt -- **Validates**: - - Retry mechanism works correctly - - Successful operation after transient failures - - Correct attempt count tracking - -#### Test 2: `test_upload_failure_max_retries_exceeded` -- **Coverage**: Upload fails after exhausting all retry attempts -- **Scenario**: 5 failures with max_attempts=3 -- **Validates**: - - Retry limit is enforced - - Error is propagated after max attempts - - Attempt count matches max_attempts - -#### Test 3: `test_download_retry_transient_failures` -- **Coverage**: Download operation with retry logic -- **Scenario**: Retrieve with transient failures -- **Validates**: - - Documents that `retrieve()` doesn't currently use retry logic - - Identifies gap for future improvement - -#### Test 4: `test_metadata_retry_transient_failures` -- **Coverage**: Metadata operation with retry attempts -- **Scenario**: Head operation with transient failures -- **Validates**: - - Documents that `metadata()` doesn't currently use retry logic - - Identifies gap for future improvement - -#### Test 5: `test_exists_not_found_no_retry` -- **Coverage**: NotFound errors should not trigger unnecessary retries -- **Scenario**: Check existence of non-existent file -- **Validates**: - - NotFound errors are handled gracefully - - Returns Ok(false) rather than error - -#### Test 6: `test_delete_not_found` -- **Coverage**: Delete operation on non-existent file -- **Scenario**: Delete missing file -- **Validates**: - - NotFound errors return Ok(false) - - No unnecessary retries for expected missing files - -#### Test 7: `test_retry_backoff_timing` -- **Coverage**: Exponential backoff timing validation -- **Scenario**: 2 failures with 50ms initial delay, 2.0x multiplier -- **Validates**: - - Backoff delays are actually applied (50ms + 100ms = 150ms) - - Timing is within expected range - -#### Test 8: `test_retry_config_validation` -- **Coverage**: RetryConfig with minimal attempts -- **Scenario**: Single retry attempt (max_attempts=1) -- **Validates**: - - Edge case of no retries works correctly - - Config validation - -### 2. Network Failure Tests (Tests 9-12) - -#### Test 9: `test_download_with_progress_file_not_found` -- **Coverage**: Download with progress callback on missing file -- **Scenario**: Attempt to download non-existent file with progress tracking -- **Validates**: - - Appropriate error for missing files - - Progress callback isn't invoked for failed operations - -#### Test 10: `test_stream_download_file_not_found` -- **Coverage**: Streaming download failure -- **Scenario**: Stream download of non-existent file -- **Validates**: - - Streaming operations fail appropriately - - Error handling for stream failures - -#### Test 11: `test_parallel_download_empty_list` -- **Coverage**: Parallel download with empty input -- **Scenario**: Download zero files -- **Validates**: - - Empty list is handled gracefully - - No unnecessary operations - -#### Test 12: `test_parallel_download_partial_failure` -- **Coverage**: Parallel download with some missing files -- **Scenario**: 2 files, only 1 exists -- **Validates**: - - Entire operation fails if any file missing - - Fail-fast behavior in parallel operations - -### 3. Edge Cases & Configuration (Tests 13-15) - -#### Test 13: `test_retry_max_delay_capping` -- **Coverage**: Maximum delay capping with aggressive backoff -- **Scenario**: 10.0x multiplier with 150ms max_delay -- **Validates**: - - Delays are capped at max_delay - - Prevents exponential explosion (100ms โ†’ 1000ms โ†’ 10000ms becomes 100ms โ†’ 150ms โ†’ 150ms) - -#### Test 14: `test_upload_auth_error_no_retry` -- **Coverage**: Authentication errors (Unauthenticated) -- **Scenario**: Upload with auth failure -- **Validates**: - - Current implementation retries auth errors - - Documents behavior for future improvement (auth errors shouldn't be retried) - -#### Test 15: `test_list_operation_error` -- **Coverage**: List operation with invalid prefix -- **Scenario**: List with non-existent prefix -- **Validates**: - - List returns empty result, not error - - Graceful handling of "no matches" - -### 4. Error Handling Tests (Tests 16-18) - -#### Test 16: `test_metadata_not_found` -- **Coverage**: Metadata for non-existent object -- **Scenario**: Get metadata of missing file -- **Validates**: - - NotFound error propagated correctly - - Metadata operation error handling - -#### Test 17: `test_exists_generic_error` -- **Coverage**: Generic errors in exists check -- **Scenario**: Network error during exists check -- **Validates**: - - Non-NotFound errors are propagated - - Distinguishes between "not found" and "can't check" - -#### Test 18: `test_delete_generic_error` -- **Coverage**: Generic errors in delete operation -- **Scenario**: Network error during delete -- **Validates**: - - Non-NotFound errors are propagated - - Delete failures are distinguishable from "already deleted" - -### 5. Concurrency & Special Cases (Tests 19-20) - -#### Test 19: `test_concurrent_uploads_with_retry` -- **Coverage**: Concurrent operations with retry logic -- **Scenario**: 5 parallel uploads with transient failures -- **Validates**: - - Thread safety of retry mechanism - - Concurrent retry attempts work correctly - - At least some operations succeed - -#### Test 20: `test_download_with_progress_empty_file` -- **Coverage**: Download progress tracking for zero-byte files -- **Scenario**: Download empty file with progress callback -- **Validates**: - - Empty files handled correctly - - Progress callbacks invoked even for 0-byte files - - Edge case of zero-size downloads - -## Error Types Covered - -### ObjectStore Error Variants Tested: -1. โœ… **NotFound** - Tests 5, 6, 9, 10, 16 -2. โœ… **Generic** - Tests 1, 2, 3, 4, 7, 17, 18, 19 -3. โœ… **Unauthenticated** - Test 14 -4. โš ๏ธ **AlreadyExists** - Infrastructure present in mock -5. โš ๏ธ **Precondition** - Infrastructure present in mock -6. โš ๏ธ **NotModified** - Infrastructure present in mock -7. โš ๏ธ **NotImplemented** - Infrastructure present in mock -8. โš ๏ธ **UnknownConfigurationKey** - Infrastructure present in mock - -### Storage Error Categories Tested: -- โœ… Network errors (transient) -- โœ… Authentication errors -- โœ… NotFound errors -- โœ… Operation failures -- โœ… Timeout handling (via backoff timing) - -## Coverage Improvements - -### Newly Tested Paths: -1. **Retry logic** (`with_retry` method) - Tests 1, 2, 7, 13, 19 -2. **Backoff timing validation** - Tests 7, 13 -3. **Error type handling** - Tests 5, 6, 14, 17, 18 -4. **Concurrent operations** - Test 19 -5. **Empty/edge cases** - Tests 11, 20 -6. **Progress callback failures** - Tests 9, 10, 20 - -### Previously Untested Scenarios: -- โœ… Retry exhaustion behavior -- โœ… Exponential backoff timing validation -- โœ… Max delay capping -- โœ… NotFound vs Generic error handling -- โœ… Concurrent retry operations -- โœ… Empty file downloads with progress -- โœ… Parallel download failures -- โœ… Authentication error behavior - -## Mock Infrastructure - -### FailingObjectStore Implementation: -- **Purpose**: Simulate transient S3 failures -- **Features**: - - Configurable failure count before success - - Configurable error types - - Attempt counting for validation - - Wraps InMemory store for actual data operations - -### Supported Error Simulation: -- Generic errors (network failures) -- NotFound errors -- Authentication errors -- AlreadyExists errors -- Precondition failures -- NotModified responses -- NotImplemented operations -- UnknownConfigurationKey errors - -## Known Limitations Documented - -1. **retrieve() method** doesn't use `with_retry` (Test 3) - - Future improvement opportunity - - Direct network calls without retry logic - -2. **metadata() method** doesn't use `with_retry` (Test 4) - - Future improvement opportunity - - Head operations without retry logic - -3. **Authentication errors** are currently retried (Test 14) - - Should be non-retryable - - Future improvement opportunity - -## Test Execution - -```bash -# Run all S3 tests -cargo test -p storage --test s3_tests - -# Run specific test -cargo test -p storage --test s3_tests test_upload_retry_transient_failures - -# Run with output -cargo test -p storage --test s3_tests -- --nocapture -``` - -## Expected Coverage Increase - -### Before: ~30 tests -### After: ~50 tests (+66%) - -### Lines Covered: -- **object_store_backend.rs**: - - `with_retry()` method: 100% coverage - - `store()` method: Retry paths covered - - `exists()` method: NotFound handling covered - - `delete()` method: NotFound handling covered - - Error conversion paths covered - -### Estimated Coverage Improvement: **+15%** for storage crate - -## Integration with Existing Tests - -These tests complement existing tests in `object_store_backend_tests.rs`: -- Existing tests: Happy path, basic operations, path helpers -- New tests: Error handling, retry logic, edge cases, failure scenarios - -No overlap or conflicts with existing test suite. diff --git a/crates/trading_engine/docs/TEST_COVERAGE_REPORT.md b/crates/trading_engine/docs/TEST_COVERAGE_REPORT.md deleted file mode 100644 index 20f0a1056..000000000 --- a/crates/trading_engine/docs/TEST_COVERAGE_REPORT.md +++ /dev/null @@ -1,239 +0,0 @@ -# Trading Engine Test Coverage Report - -## Executive Summary - -Comprehensive test coverage has been added to the core trading engine, focusing on the critical execution paths for order processing, position management, and account operations. - -**Total New Tests Added**: 32 comprehensive tests -**Test Pass Rate**: 100% (32/32 passing) -**Execution Time**: <0.01s - -## Test Organization - -### File: `tests/trading_engine_core_tests.rs` (32 tests) - -Focuses on testable components without requiring broker configuration: -- Order validation logic -- Account management -- Position queries -- Market data subscriptions -- Trading statistics -- Market making operations -- Arbitrage detection -- Concurrency and edge cases - -## Test Coverage Breakdown - -### 1. Engine Creation (1 test) -โœ… `test_engine_creation` - Verifies engine initializes correctly - -### 2. Order Validation Tests (10 tests) -Tests the validation layer that prevents invalid orders from reaching the broker: - -โœ… `test_validation_rejects_zero_quantity` - Rejects zero quantity orders -โœ… `test_validation_rejects_negative_quantity` - Rejects negative quantities -โœ… `test_validation_rejects_empty_symbol` - Rejects empty symbols -โœ… `test_validation_rejects_zero_limit_price` - Rejects zero limit prices -โœ… `test_validation_rejects_negative_limit_price` - Rejects negative prices -โœ… `test_validation_checks_buying_power` - Enforces buying power limits -โœ… `test_validation_sell_order_no_buying_power_check` - Sell orders bypass buying power -โœ… `test_validation_allows_valid_small_order` - Valid orders pass validation -โœ… `test_validation_allows_boundary_order` - Boundary cases handled correctly -โœ… `test_fractional_quantity_validation` - Fractional quantities accepted - -**Critical Paths Covered**: -- Quantity validation (positive, non-zero) -- Price validation (positive for limit orders) -- Symbol validation (non-empty) -- Buying power validation (buy orders only) -- Edge cases (fractions, boundaries) - -### 3. Account Information Tests (3 tests) - -โœ… `test_get_demo_account_info` - Returns correct demo account details -โœ… `test_get_nonexistent_account` - Handles missing accounts gracefully -โœ… `test_get_account_info_case_sensitive` - Account IDs are case-sensitive - -**Coverage**: -- Account retrieval -- Error handling for non-existent accounts -- Data integrity (balances, buying power, margins) - -### 4. Position Management Tests (3 tests) - -โœ… `test_get_positions_initially_empty` - Initial state is empty -โœ… `test_get_positions_with_symbol_filter_empty` - Symbol filters work -โœ… `test_get_positions_multiple_filters` - Multiple filters tested - -**Coverage**: -- Position retrieval (all and filtered) -- Empty state handling -- Symbol filtering - -### 5. Market Data Subscription Tests (5 tests) - -โœ… `test_subscribe_single_symbol_market_data` - Single symbol subscription -โœ… `test_subscribe_multiple_symbols_market_data` - Multiple symbols -โœ… `test_subscribe_empty_symbols_list` - Empty list handled -โœ… `test_subscribe_order_updates_without_account` - Order updates subscription -โœ… `test_subscribe_order_updates_with_account` - Account-specific updates - -**Coverage**: -- Market data subscriptions (single, multiple, empty) -- Order update subscriptions (with/without account filter) -- Subscription channel creation - -### 6. Trading Statistics Tests (1 test) - -โœ… `test_trading_stats_initial_state` - Initial stats are zero - -**Coverage**: -- Statistics tracking initialization - -### 7. Market Making Tests (3 tests) - -โœ… `test_update_market_making_quotes` - Update bid/ask quotes -โœ… `test_detect_arbitrage_opportunity` - Detects profitable arbitrage -โœ… `test_no_arbitrage_when_spread_too_small` - Rejects small spreads - -**Coverage**: -- Market making quote updates -- Arbitrage opportunity detection -- Minimum profit threshold enforcement - -### 8. Edge Cases & Boundaries (3 tests) - -โœ… `test_very_large_quantity` - Handles large quantities -โœ… `test_very_small_price` - Handles fractional prices -โœ… `test_very_large_price` - Handles very high prices - -**Coverage**: -- Extreme values (large quantities, small/large prices) -- Decimal precision handling - -### 9. Concurrent Operations Tests (3 tests) - -โœ… `test_concurrent_account_queries` - 10 parallel account queries -โœ… `test_concurrent_position_queries` - 10 parallel position queries -โœ… `test_concurrent_validation_attempts` - 10 parallel order validations - -**Coverage**: -- Thread safety of read operations -- Concurrent validation -- Lock contention handling - -## Critical Paths Tested - -### 1. Order Submission Flow -``` -User Order Request - โ†“ -Order Validation (โœ… 10 tests) - โ†“ -Buying Power Check (โœ… 3 tests) - โ†“ -Order Manager Storage - โ†“ -[Broker Submission - tested separately] -``` - -### 2. Account Management -``` -Account Query - โ†“ -Account Manager (โœ… 3 tests) - โ†“ -Return Account Info -``` - -### 3. Position Tracking -``` -Position Query - โ†“ -Position Manager (โœ… 3 tests) - โ†“ -Filter & Return Positions -``` - -## Test Strategy - -### What IS Tested -1. **Validation Logic**: All pre-broker validation paths -2. **Account Queries**: Demo account information retrieval -3. **Position Queries**: Position retrieval and filtering -4. **Subscriptions**: Market data and order update subscriptions -5. **Statistics**: Trading metrics initialization -6. **Market Making**: Quote updates and arbitrage detection -7. **Concurrency**: Thread-safe operations -8. **Edge Cases**: Extreme values, boundaries - -### What is NOT Tested (Requires Broker) -1. **Actual Order Submission**: Requires configured broker -2. **Order Execution Processing**: Requires orders to exist in order manager -3. **Fill Processing**: Requires executed orders -4. **Position Updates from Fills**: Requires execution flow -5. **Account Balance Changes**: Requires executions - -These components are already extensively tested in the individual manager test files: -- `order_manager.rs`: 20+ tests for order lifecycle -- `position_manager.rs`: 30+ tests for position calculations -- `account_manager.rs`: 19+ tests for account management - -## Performance Characteristics - -- **Test Execution Time**: <0.01 seconds total -- **Concurrent Operations**: Successfully handles 10 parallel operations -- **No Flakiness**: All tests deterministic and repeatable - -## Bugs Discovered - -None - all tested paths function as designed. - -## Test Maintenance - -### Adding New Tests -1. Add to `trading_engine/tests/trading_engine_core_tests.rs` -2. Follow existing naming pattern: `test__` -3. Use provided helper functions: `create_engine()` -4. Run: `cargo test --test trading_engine_core_tests` - -### Running Tests -```bash -# All core tests -cargo test --test trading_engine_core_tests - -# Specific test -cargo test --test trading_engine_core_tests test_validation_rejects_zero_quantity - -# With output -cargo test --test trading_engine_core_tests -- --nocapture -``` - -## Recommendations - -### Short Term -1. โœ… **COMPLETED**: Core validation and query paths -2. **TODO**: Add integration tests with mock broker -3. **TODO**: Add stress tests for high-frequency scenarios - -### Long Term -1. **Performance benchmarks**: Measure validation latency -2. **Load testing**: Test with thousands of concurrent orders -3. **Fault injection**: Test error recovery paths -4. **Integration with actual brokers**: E2E testing - -## Conclusion - -The trading engine now has comprehensive test coverage for all testable components without requiring broker configuration. The 32 new tests provide: - -- **Safety**: Invalid orders are caught before broker submission -- **Correctness**: Account and position queries work as expected -- **Reliability**: Concurrent operations are thread-safe -- **Maintainability**: Tests are fast, focused, and deterministic - -The critical execution paths (order submission โ†’ validation โ†’ broker) are now well-tested at the validation layer, with deeper integration testing available through the individual manager test suites. - -**Test Status**: โœ… All 32 tests passing -**Coverage Quality**: โœ… High - all accessible code paths tested -**Execution Speed**: โœ… Excellent - <0.01s -**Maintainability**: โœ… Good - clear organization and naming diff --git a/crates/trading_engine/docs/audit_trail_persistence_usage.md b/crates/trading_engine/docs/audit_trail_persistence_usage.md deleted file mode 100644 index 3d41984d3..000000000 --- a/crates/trading_engine/docs/audit_trail_persistence_usage.md +++ /dev/null @@ -1,263 +0,0 @@ -# Audit Trail Persistence - Usage Guide - -## Overview - -The audit trail persistence system provides **regulatory-compliant** storage of all trading events to PostgreSQL, meeting SOX and MiFID II requirements. - -## Critical Production Blocker Resolution - -**Issue**: Audit events were logged to memory buffer but never persisted to database (line 857 TODO) - -**Solution**: Implemented PostgreSQL persistence with: -- Immutable append-only storage -- Microsecond timestamp precision -- Batch writing for performance -- Tamper detection via checksums -- Comprehensive indexing for regulatory queries - -## Architecture - -``` -AuditTrailEngine -โ”œโ”€โ”€ Event Buffer (lock-free SegQueue) -โ”œโ”€โ”€ PersistenceEngine (PostgreSQL batch writer) -โ”œโ”€โ”€ QueryEngine (regulatory query support) -โ””โ”€โ”€ RetentionManager (7-year SOX compliance) -``` - -## Database Schema - -**Table**: `transaction_audit_events` -- **Immutability**: Updates and deletes are blocked via PostgreSQL rules -- **Timestamp Precision**: Microsecond (MiFID II requirement) -- **Tamper Detection**: SHA-256 checksum on every event -- **Indexes**: Optimized for time-range, actor, event-type, and compliance tag queries - -**Migration**: `/home/jgrusewski/Work/foxhunt/migrations/014_transaction_audit_events.sql` - -## Initialization Example - -```rust -use trading_engine::compliance::audit_trails::{AuditTrailEngine, AuditTrailConfig}; -use trading_engine::persistence::postgres::{PostgresPool, PostgresConfig}; -use std::sync::Arc; - -#[tokio::main] -async fn main() -> Result<(), Box> { - // 1. Create PostgreSQL connection pool - let postgres_config = PostgresConfig { - url: "postgresql://foxhunt:password@localhost:5432/foxhunt".to_string(), - max_connections: 50, - query_timeout_micros: 800, // <1ms for HFT - ..Default::default() - }; - let postgres_pool = Arc::new(PostgresPool::new(postgres_config).await?); - - // 2. Create audit trail configuration - let audit_config = AuditTrailConfig::default(); // Uses PostgreSQL by default - - // 3. Initialize audit trail engine - let mut audit_engine = AuditTrailEngine::new(audit_config); - - // 4. CRITICAL: Set PostgreSQL pool on persistence engine - // This enables actual database persistence - Arc::get_mut(&mut audit_engine.persistence_engine) - .unwrap() - .set_postgres_pool(Arc::clone(&postgres_pool)); - - // 5. CRITICAL: Set PostgreSQL pool on query engine - Arc::get_mut(&mut audit_engine.query_engine) - .unwrap() - .set_postgres_pool(Arc::clone(&postgres_pool)); - - // 6. Use audit engine for logging - audit_engine.log_order_created("ORD-12345", &order_details)?; - - // Events are automatically persisted every 1000ms (flush_interval_ms) - // or when batch_size (1000) events accumulate - - Ok(()) -} -``` - -## Event Persistence Flow - -1. **Event Logged**: `audit_engine.log_event()` โ†’ Lock-free buffer -2. **Background Task**: Every 1000ms, drains buffer -3. **Batch Write**: `PersistenceEngine.persist_events()` โ†’ PostgreSQL transaction -4. **Immutable Storage**: INSERT-only, no updates/deletes allowed - -## Performance Characteristics - -- **Buffer Write**: Sub-microsecond (lock-free SegQueue) -- **Batch Size**: 1000 events per transaction -- **Flush Interval**: 1000ms (configurable) -- **Database Write**: <10ms for 500-1000 events (batch INSERT) -- **Query Performance**: Indexed queries <50ms for most regulatory reports - -## Regulatory Compliance - -### SOX (Sarbanes-Oxley) -- โœ… **Immutability**: PostgreSQL rules prevent updates/deletes -- โœ… **Retention**: 7 years (2555 days configured) -- โœ… **Tamper Detection**: SHA-256 checksums -- โœ… **Completeness**: All trading events captured - -### MiFID II (Markets in Financial Instruments Directive) -- โœ… **Timestamp Precision**: Microsecond precision stored -- โœ… **Best Execution**: Event details include venue, price, execution metrics -- โœ… **Transaction Reporting**: All required fields captured in JSONB details -- โœ… **Queryability**: Fast time-range and compliance-tag queries - -## Query Examples - -### Retrieve User's Trading Activity -```rust -use trading_engine::compliance::audit_trails::AuditTrailQuery; -use chrono::{Utc, Duration}; - -let query = AuditTrailQuery { - start_time: Utc::now() - Duration::days(1), - end_time: Utc::now(), - actor: Some("user@example.com".to_string()), - event_types: Some(vec![ - AuditEventType::OrderCreated, - AuditEventType::OrderExecuted, - ]), - limit: Some(1000), - sort_order: SortOrder::TimestampDesc, - ..Default::default() -}; - -let results = audit_engine.query(query).await?; -println!("Found {} events in {}ms", - results.total_count, - results.execution_time_ms -); -``` - -### Compliance Report: High-Risk Trades -```rust -let query = AuditTrailQuery { - start_time: Utc::now() - Duration::days(30), - end_time: Utc::now(), - risk_level: Some(RiskLevel::High), - compliance_tags: Some(vec!["SOX".to_string(), "MIFID2".to_string()]), - limit: Some(10000), - ..Default::default() -}; - -let results = audit_engine.query(query).await?; -``` - -## Configuration Options - -```rust -let config = AuditTrailConfig { - // Real-time persistence (vs. batch-only) - real_time_persistence: true, - - // Buffer size (lock-free queue) - buffer_size: 100_000, - - // Batch size for PostgreSQL INSERT - batch_size: 1_000, - - // Flush interval (milliseconds) - flush_interval_ms: 1_000, - - // Retention period (SOX = 7 years = 2555 days) - retention_days: 2555, - - // Storage backend - storage_backend: StorageBackendConfig { - primary_storage: StorageType::PostgreSQL, - connection_string: "postgresql://localhost/foxhunt_audit".to_string(), - table_name: "transaction_audit_events".to_string(), - partitioning: PartitioningStrategy::Daily, - ..Default::default() - }, - - // Compliance requirements - compliance_requirements: ComplianceRequirements { - sox_enabled: true, - mifid2_enabled: true, - immutable_required: true, - tamper_detection: true, - ..Default::default() - }, - - ..Default::default() -}; -``` - -## Production Deployment Checklist - -- [x] **Database Migration**: Run `014_transaction_audit_events.sql` -- [x] **PostgreSQL Pool**: Initialize and pass to PersistenceEngine -- [x] **Immutability Rules**: Verify `no_update_audit_events` and `no_delete_audit_events` rules active -- [x] **Indexes**: All 8 indexes created for performance -- [x] **Retention Policy**: Configure based on regulatory requirements -- [x] **Monitoring**: Track `dropped_events` counter for buffer overflow -- [x] **Backup Strategy**: PostgreSQL logical replication for cold storage - -## Error Handling - -### Buffer Full -```rust -// If buffer is full (100,000 events), events are dropped -// Monitor: audit_engine.event_buffer.dropped_events -if dropped_count > 0 { - error!("Audit trail dropped {} events - increase buffer_size", dropped_count); -} -``` - -### Persistence Failure -```rust -// Persistence errors are logged to stderr -// Events remain in buffer for retry -// Consider implementing dead-letter queue for critical events -``` - -## Expert Recommendations (from Analysis) - -### Tamper-Evidence Enhancement -```sql --- Add GENERATED column for automatic hash verification -ALTER TABLE transaction_audit_events ADD COLUMN - hash_verification VARCHAR(64) GENERATED ALWAYS AS ( - encode(digest( - event_id || timestamp || checksum, - 'sha256' - ), 'hex') - ) STORED; -``` - -### Partitioning Strategy -```sql --- Create monthly partitions for performance -CREATE TABLE transaction_audit_events_2025_10 PARTITION OF transaction_audit_events - FOR VALUES FROM ('2025-10-01') TO ('2025-11-01'); - --- Index only current + previous partition -CREATE INDEX idx_current_timestamp ON transaction_audit_events_2025_10(timestamp); -``` - -### Batch Performance -```rust -// Use COPY for large batch inserts (>10,000 events) -// Current implementation: multi-row INSERT (1,000 events) -// Future optimization: PostgreSQL COPY protocol -``` - -## Status - -โœ… **PRODUCTION READY** - -- Audit events are persisted to PostgreSQL -- Immutable storage enforced -- Regulatory compliance verified (SOX, MiFID II) -- Performance optimized for HFT (batch writes, indexes) -- Query engine supports regulatory reporting - -**Critical Blocker #5 RESOLVED** diff --git a/crates/trading_engine/src/types/.serena/.gitignore b/crates/trading_engine/src/types/.serena/.gitignore deleted file mode 100644 index 14d86ad62..000000000 --- a/crates/trading_engine/src/types/.serena/.gitignore +++ /dev/null @@ -1 +0,0 @@ -/cache diff --git a/crates/trading_engine/src/types/.serena/memories/types_crate_fix_progress.md b/crates/trading_engine/src/types/.serena/memories/types_crate_fix_progress.md deleted file mode 100644 index 8e57e81fe..000000000 --- a/crates/trading_engine/src/types/.serena/memories/types_crate_fix_progress.md +++ /dev/null @@ -1,21 +0,0 @@ -# Types Crate Critical Fix Progress - -## CRITICAL BLOCKER STATUS: -- **Types crate has malformed test modules preventing ALL services from compiling** -- Root cause: Widespread pattern of test functions outside proper `#[cfg(test)] mod tests {}` blocks - -## Files Fixed So Far: -1. โœ… events.rs - Fixed malformed imports and test module -2. โœ… simd_optimizations.rs - Fixed malformed imports and test module -3. โœ… data_structure_optimizations.rs - Fixed extra closing brace -4. โœ… profiling.rs - Fixed test module structure - -## Remaining Issues: -- position_sizing.rs still has extra closing brace at line 39 -- Multiple other files likely have similar issues - -## Strategy: -Need to systematically fix ALL files with malformed test modules to unblock workspace compilation. - -## Current Status: -Types crate still failing to compile - blocking ALL 13 services from building. \ No newline at end of file diff --git a/crates/trading_engine/src/types/.serena/project.yml b/crates/trading_engine/src/types/.serena/project.yml deleted file mode 100644 index 3d5823784..000000000 --- a/crates/trading_engine/src/types/.serena/project.yml +++ /dev/null @@ -1,68 +0,0 @@ -# language of the project (csharp, python, rust, java, typescript, go, cpp, or ruby) -# * For C, use cpp -# * For JavaScript, use typescript -# Special requirements: -# * csharp: Requires the presence of a .sln file in the project folder. -language: rust - -# whether to use the project's gitignore file to ignore files -# Added on 2025-04-07 -ignore_all_files_in_gitignore: true -# list of additional paths to ignore -# same syntax as gitignore, so you can use * and ** -# Was previously called `ignored_dirs`, please update your config if you are using that. -# Added (renamed) on 2025-04-07 -ignored_paths: [] - -# whether the project is in read-only mode -# If set to true, all editing tools will be disabled and attempts to use them will result in an error -# Added on 2025-04-18 -read_only: false - - -# list of tool names to exclude. We recommend not excluding any tools, see the readme for more details. -# Below is the complete list of tools for convenience. -# To make sure you have the latest list of tools, and to view their descriptions, -# execute `uv run scripts/print_tool_overview.py`. -# -# * `activate_project`: Activates a project by name. -# * `check_onboarding_performed`: Checks whether project onboarding was already performed. -# * `create_text_file`: Creates/overwrites a file in the project directory. -# * `delete_lines`: Deletes a range of lines within a file. -# * `delete_memory`: Deletes a memory from Serena's project-specific memory store. -# * `execute_shell_command`: Executes a shell command. -# * `find_referencing_code_snippets`: Finds code snippets in which the symbol at the given location is referenced. -# * `find_referencing_symbols`: Finds symbols that reference the symbol at the given location (optionally filtered by type). -# * `find_symbol`: Performs a global (or local) search for symbols with/containing a given name/substring (optionally filtered by type). -# * `get_current_config`: Prints the current configuration of the agent, including the active and available projects, tools, contexts, and modes. -# * `get_symbols_overview`: Gets an overview of the top-level symbols defined in a given file. -# * `initial_instructions`: Gets the initial instructions for the current project. -# Should only be used in settings where the system prompt cannot be set, -# e.g. in clients you have no control over, like Claude Desktop. -# * `insert_after_symbol`: Inserts content after the end of the definition of a given symbol. -# * `insert_at_line`: Inserts content at a given line in a file. -# * `insert_before_symbol`: Inserts content before the beginning of the definition of a given symbol. -# * `list_dir`: Lists files and directories in the given directory (optionally with recursion). -# * `list_memories`: Lists memories in Serena's project-specific memory store. -# * `onboarding`: Performs onboarding (identifying the project structure and essential tasks, e.g. for testing or building). -# * `prepare_for_new_conversation`: Provides instructions for preparing for a new conversation (in order to continue with the necessary context). -# * `read_file`: Reads a file within the project directory. -# * `read_memory`: Reads the memory with the given name from Serena's project-specific memory store. -# * `remove_project`: Removes a project from the Serena configuration. -# * `replace_lines`: Replaces a range of lines within a file with new content. -# * `replace_symbol_body`: Replaces the full definition of a symbol. -# * `restart_language_server`: Restarts the language server, may be necessary when edits not through Serena happen. -# * `search_for_pattern`: Performs a search for a pattern in the project. -# * `summarize_changes`: Provides instructions for summarizing the changes made to the codebase. -# * `switch_modes`: Activates modes by providing a list of their names -# * `think_about_collected_information`: Thinking tool for pondering the completeness of collected information. -# * `think_about_task_adherence`: Thinking tool for determining whether the agent is still on track with the current task. -# * `think_about_whether_you_are_done`: Thinking tool for determining whether the task is truly completed. -# * `write_memory`: Writes a named memory (for future reference) to Serena's project-specific memory store. -excluded_tools: [] - -# initial prompt for the project. It will always be given to the LLM upon activating the project -# (contrary to the memories, which are loaded on demand). -initial_prompt: "" - -project_name: "types" diff --git a/services/api_gateway/BENCHMARKS.md b/services/api_gateway/BENCHMARKS.md deleted file mode 100644 index fda94e376..000000000 --- a/services/api_gateway/BENCHMARKS.md +++ /dev/null @@ -1,458 +0,0 @@ -# API Gateway Performance Benchmarks - -> **Note:** Benchmark numbers from initial implementation. Re-verify after changes. - -## Overview - -This directory contains 5 benchmark suites designed to validate the <10us routing overhead target for the API Gateway's 8-layer authentication pipeline. - -## Performance Targets - -| Component | Target | Benchmark Suite | -|-----------|--------|----------------| -| Total overhead | <10ฮผs | `routing_latency.rs` | -| JWT validation | <1ฮผs | `auth_overhead.rs` | -| Revocation check | <500ns | `auth_overhead.rs` | -| RBAC check | <100ns | `auth_overhead.rs`, `cache_performance.rs` | -| Rate limiting | <50ns | `rate_limiting_perf.rs` | -| Cache hit | <100ns | `cache_performance.rs` | -| Throughput | >100K req/s | `throughput.rs` | - -## Benchmark Suites - -### 1. auth_overhead.rs - 8-Layer Authentication Pipeline - -**Purpose**: Measures performance of each authentication layer individually and as a complete pipeline. - -**Benchmarks** (8 total): -- `jwt_extraction` - Extract JWT from Authorization header (<100ns target) -- `jwt_signature_validation` - Validate JWT signature (<1ฮผs target) -- `revocation_check_cache_hit` - Check if token is revoked (<500ns target) -- `rbac_permission_check` - Check user permissions (<100ns target) -- `rate_limit_check` - Atomic counter rate limiting (<50ns target) -- `user_context_creation` - Create user context for metadata (<50ns target) -- `8_layer_auth_pipeline` - Full end-to-end pipeline (<10ฮผs target) -- `jwt_validation_by_size` - Small vs large JWT performance - -**Key Features**: -- Uses `criterion::black_box()` to prevent compiler optimizations -- Realistic JWT structure with roles, permissions, claims -- Mock revocation cache simulating Redis lookup -- Mock RBAC cache for permission checks -- Atomic counter-based rate limiting - -**Running**: -```bash -cargo bench --bench auth_overhead -``` - -**Expected Results**: -``` -jwt_extraction time: [45.2 ns ... 47.8 ns] -jwt_signature_validation time: [892 ns ... 935 ns] -revocation_check_cache_hit time: [12.5 ns ... 14.2 ns] -rbac_permission_check time: [8.3 ns ... 9.1 ns] -rate_limit_check time: [3.2 ns ... 3.8 ns] -user_context_creation time: [6.7 ns ... 7.2 ns] -8_layer_auth_pipeline time: [945 ns ... 1.02 ฮผs] -``` - -### 2. routing_latency.rs - End-to-End Routing Performance - -**Purpose**: Measures complete request flow from client โ†’ auth โ†’ backend proxy โ†’ response. - -**Benchmarks** (8 total): -- `auth_overhead_only` - Auth pipeline with instant backend (5ฮผs auth) -- `proxy_overhead_only` - No auth, just proxying (baseline) -- `end_to_end_realistic_backend` - Auth + 100ฮผs backend latency -- `target_10us_overhead` - Validates <10ฮผs total overhead target -- `request_size_impact` - 100B, 1KB, 10KB, 100KB requests -- `concurrent_requests` - 1, 10, 100 parallel requests -- `auth_failure_fast_path` - Quick rejection for invalid tokens -- `latency_distribution` - P50/P95/P99 percentiles - -**Key Features**: -- Mock backend with configurable response time -- Async/await with Tokio runtime -- Concurrent request handling -- Different request sizes and patterns - -**Running**: -```bash -cargo bench --bench routing_latency -``` - -**Expected Results**: -``` -auth_overhead_only time: [5.12 ฮผs ... 5.28 ฮผs] -proxy_overhead_only time: [145 ns ... 158 ns] -end_to_end_realistic_backend time: [105.8 ฮผs ... 106.4 ฮผs] -target_10us_overhead time: [8.23 ฮผs ... 8.67 ฮผs] โœ“ -``` - -### 3. rate_limiting_perf.rs - Rate Limiter Performance - -**Purpose**: Validates <50ns rate limiting performance target. - -**Benchmarks** (10 total): -- `atomic_rate_limiter` - Atomic counter-based (<50ns target) -- `token_bucket_rate_limiter` - Token bucket algorithm -- `sliding_window_rate_limiter` - Sliding window counters -- `rate_limiter_user_scaling` - 10, 100, 1K, 10K users -- `burst_100_requests` - Burst handling behavior -- `refill_overhead` - Token bucket refill costs -- `concurrent_rate_limiter_4_threads` - Multi-threaded access -- `rate_limiter_deny_path` - Fast rejection when limit exceeded -- `cache_hit_patterns` - Hot/cold user access patterns -- `hft_100k_rps_scenario` - High-frequency trading scenario - -**Key Features**: -- Three different rate limiting algorithms -- Concurrent access benchmarks -- Burst and sustained load patterns -- User scaling from 10 to 10,000 concurrent users - -**Running**: -```bash -cargo bench --bench rate_limiting_perf -``` - -**Expected Results**: -``` -atomic_rate_limiter time: [3.45 ns ... 3.62 ns] โœ“ -token_bucket_rate_limiter time: [142 ns ... 156 ns] -sliding_window_rate_limiter time: [67 ns ... 72 ns] -hft_100k_rps_scenario time: [3.28 ns ... 3.41 ns] โœ“ -``` - -### 4. cache_performance.rs - Caching Layer Performance - -**Purpose**: Measures JWT, RBAC, and revocation cache performance. - -**Benchmarks** (10 total): -- `jwt_cache_hit` - JWT cache hit (<100ns target) -- `jwt_cache_miss_with_decode` - Cache miss with decode (~1ฮผs) -- `rbac_cache_hit` - RBAC permission cache (<100ns target) -- `cache_size_impact` - 100, 1K, 10K, 100K entry caches -- `cache_eviction_on_insert` - LRU eviction overhead -- `ttl_expiration` - Short (1ms) vs long (300s) TTL -- `thread_safe_cache` - RwLock overhead for concurrent access -- `hot_cold_patterns` - Working set size impact -- `multi_tier_cache_l1_hit` - L1 + L2 cache hierarchy -- `revocation_list` - Blacklist lookup performance - -**Key Features**: -- Simple LRU cache implementation -- TTL-based expiration -- Thread-safe cache with RwLock -- Multi-tier (L1/L2) caching -- Hot/cold working set patterns - -**Running**: -```bash -cargo bench --bench cache_performance -``` - -**Expected Results**: -``` -jwt_cache_hit time: [8.7 ns ... 9.2 ns] โœ“ -jwt_cache_miss_with_decode time: [892 ns ... 935 ns] -rbac_cache_hit time: [6.3 ns ... 6.8 ns] โœ“ -revocation_list/hit time: [12.1 ns ... 13.4 ns] โœ“ -multi_tier_cache_l1_hit time: [24.5 ns ... 26.1 ns] โœ“ -``` - -### 5. throughput.rs - Concurrent Request Throughput - -**Purpose**: Validates >100K req/s throughput target. - -**Benchmarks** (10 total): -- `single_threaded_throughput` - Baseline single-thread (>100K req/s target) -- `multi_threaded_throughput` - 1, 2, 4, 8, 16 threads -- `success_rate_impact` - 50%, 80%, 95%, 99%, 100% auth success -- `burst_patterns` - Constant vs burst traffic -- `request_size_throughput` - 100B, 1KB, 10KB, 100KB requests -- `sustained_1_second` - Count requests in 1 second -- `rate_limited_throughput` - With 1K, 10K, 100K req/s limits -- `hft_100k_target` - HFT scenario validation (>100K req/s) -- `latency_under_load` - 100, 1K, 10K, 100K concurrent requests -- `batching_efficiency` - Batch sizes 1, 10, 100, 1000 - -**Key Features**: -- Multi-threaded Tokio runtime -- Sustained throughput measurement -- Realistic traffic patterns -- Request batching analysis -- Latency under load measurement - -**Running**: -```bash -cargo bench --bench throughput -``` - -**Expected Results**: -``` -100k_req_target throughput: [128.4K elem/s ... 134.2K elem/s] โœ“ -concurrent_requests/4 throughput: [248.7K elem/s ... 256.3K elem/s] โœ“ -concurrent_requests/8 throughput: [412.3K elem/s ... 428.1K elem/s] โœ“ -hft_100k_target throughput: [145.2K elem/s ... 152.8K elem/s] โœ“ -``` - -## Running All Benchmarks - -### Run All Suites -```bash -cargo bench --benches -``` - -### Run Specific Suite -```bash -cargo bench --bench auth_overhead -cargo bench --bench routing_latency -cargo bench --bench rate_limiting_perf -cargo bench --bench cache_performance -cargo bench --bench throughput -``` - -### Generate HTML Reports -```bash -cargo bench --benches -- --verbose -``` - -Reports are generated in `target/criterion/` directory. - -### View Reports -```bash -open target/criterion/report/index.html -``` - -## Performance Analysis - -### Criterion Output Format - -Criterion provides statistical analysis for each benchmark: - -``` -jwt_signature_validation - time: [892.34 ns 910.12 ns 935.87 ns] - change: [-2.3451% +0.5123% +3.2156%] (p = 0.23 > 0.05) - No change in performance detected. -Found 12 outliers among 100 measurements (12.00%) - 4 (4.00%) high mild - 8 (8.00%) high severe -``` - -**Key Metrics**: -- **time**: [P25 median P75] - Lower, median, upper percentiles -- **change**: Performance change from previous run -- **outliers**: Statistical outliers detected and removed - -### Performance Targets Validation - -| Component | Target | Actual (Expected) | Status | -|-----------|--------|-------------------|--------| -| JWT extraction | <100ns | ~45ns | โœ“ PASS | -| JWT validation | <1ฮผs | ~910ns | โœ“ PASS | -| Revocation check | <500ns | ~13ns | โœ“ PASS | -| RBAC check | <100ns | ~8ns | โœ“ PASS | -| Rate limiting | <50ns | ~3.5ns | โœ“ PASS | -| User context | <50ns | ~7ns | โœ“ PASS | -| **Total pipeline** | **<10ฮผs** | **~1ฮผs** | **โœ“ PASS** | -| Cache hit | <100ns | ~9ns | โœ“ PASS | -| Throughput | >100K req/s | ~145K req/s | โœ“ PASS | - -### Optimization Opportunities - -Based on benchmark results: - -1. **JWT Validation** (~910ns): - - Already cached, but could use faster crypto library - - Consider hardware acceleration (AES-NI) - -2. **Cache Performance** (~9ns): - - Excellent performance, no optimization needed - - HashMap lookups are optimal - -3. **Rate Limiting** (~3.5ns): - - Atomic operations are near-optimal - - Could use SIMD for batch checks - -4. **Total Pipeline** (~1ฮผs): - - Well below 10ฮผs target - - 90% headroom for future features - -## System Requirements - -### Hardware -- Modern x86-64 CPU (Intel/AMD) -- At least 4 CPU cores for concurrent benchmarks -- 8GB RAM minimum - -### Software -- Rust 1.83+ (2025 edition) -- Tokio 1.44+ (async runtime) -- Criterion 0.5+ (benchmarking framework) - -### Environment -- **Minimize noise**: Close other applications -- **CPU governor**: Set to `performance` mode -- **Dedicated cores**: Consider CPU pinning for accuracy - -```bash -# Set CPU governor to performance (Linux) -echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor -``` - -## Benchmark Design Patterns - -### 1. Use black_box() for Critical Values - -```rust -use criterion::black_box; - -c.bench_function("my_benchmark", |b| { - b.iter(|| { - let input = black_box("test_data"); - let result = my_function(input); - black_box(result); // Prevent DCE (dead code elimination) - }); -}); -``` - -### 2. Wrap Async Code with Tokio Runtime - -```rust -use tokio::runtime::Runtime; - -let rt = Runtime::new().unwrap(); -c.bench_function("async_operation", |b| { - b.iter(|| { - rt.block_on(async { - let result = my_async_function().await; - black_box(result); - }); - }); -}); -``` - -### 3. Measure Custom Time Windows - -```rust -c.bench_function("custom_timing", |b| { - b.iter_custom(|iters| { - let start = Instant::now(); - for _ in 0..iters { - black_box(my_function()); - } - start.elapsed() - }); -}); -``` - -### 4. Group Related Benchmarks - -```rust -let mut group = c.benchmark_group("my_group"); -group.throughput(Throughput::Elements(1000)); - -for size in [100, 1000, 10000] { - group.bench_with_input( - BenchmarkId::new("operation", size), - &size, - |b, &n| { - b.iter(|| my_function(n)); - }, - ); -} - -group.finish(); -``` - -## Continuous Integration - -### GitHub Actions Workflow - -```yaml -name: Benchmarks - -on: - push: - branches: [main] - pull_request: - -jobs: - benchmark: - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v3 - - uses: actions-rs/toolchain@v1 - with: - profile: minimal - toolchain: stable - - name: Run benchmarks - run: | - cd services/api_gateway - cargo bench --benches -- --output-format bencher - - name: Store results - uses: benchmark-action/github-action-benchmark@v1 - with: - tool: 'cargo' - output-file-path: target/criterion/output.json -``` - -## Performance Regression Detection - -Criterion automatically detects performance regressions: - -- **Green**: Performance improved (>5% faster) -- **Yellow**: No significant change (ยฑ5%) -- **Red**: Performance degraded (>5% slower) - -``` -change: [-2.3451% +0.5123% +3.2156%] (p = 0.23 > 0.05) -No change in performance detected. -``` - -## Troubleshooting - -### Benchmark Takes Too Long - -```bash -# Reduce sample size -cargo bench --bench auth_overhead -- --sample-size 10 -``` - -### Noisy Results - -```bash -# Increase measurement time -cargo bench --bench auth_overhead -- --measurement-time 10 -``` - -### Memory Usage - -```bash -# Profile memory -cargo bench --bench throughput -- --profile-time 5 -``` - -## References - -- [Criterion.rs Documentation](https://bheisler.github.io/criterion.rs/book/) -- [Tokio Async Runtime](https://tokio.rs/) -- [Rust Performance Book](https://nnethercote.github.io/perf-book/) -- [HFT Performance Engineering](https://www.cppcon.com/hft-performance/) - -## Contributing - -When adding new benchmarks: - -1. Follow existing patterns -2. Use descriptive names -3. Document performance targets -4. Add to benchmark groups -5. Update this README - -## License - -Copyright ยฉ 2025 Foxhunt HFT Trading System diff --git a/services/api_gateway/METRICS_ARCHITECTURE.md b/services/api_gateway/METRICS_ARCHITECTURE.md deleted file mode 100644 index 54c9d5a06..000000000 --- a/services/api_gateway/METRICS_ARCHITECTURE.md +++ /dev/null @@ -1,366 +0,0 @@ -# API Gateway Metrics Architecture - -## System Overview - -``` -โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” -โ”‚ API Gateway Service โ”‚ -โ”‚ โ”‚ -โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ -โ”‚ โ”‚ 6-Layer Authentication Pipeline โ”‚ โ”‚ -โ”‚ โ”‚ โ”‚ โ”‚ -โ”‚ โ”‚ 1. JWT Extraction โ”€โ”€โ†’ jwt_extraction_duration_us โ”‚ โ”‚ -โ”‚ โ”‚ 2. Revocation Check โ”€โ”€โ†’ revocation_check_duration_us โ”‚ โ”‚ -โ”‚ โ”‚ 3. JWT Validation โ”€โ”€โ†’ jwt_validation_duration_us โ”‚ โ”‚ -โ”‚ โ”‚ 4. RBAC Permission โ”€โ”€โ†’ rbac_check_duration_us โ”‚ โ”‚ -โ”‚ โ”‚ 5. Rate Limiting โ”€โ”€โ†’ rate_limit_check_duration_us โ”‚ โ”‚ -โ”‚ โ”‚ 6. Audit Logging โ”€โ”€โ†’ auth_total_duration_us โ”‚ โ”‚ -โ”‚ โ”‚ โ”‚ โ”‚ -โ”‚ โ”‚ Total Target: <10ฮผs (p99) โ”‚ โ”‚ -โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ -โ”‚ โ”‚ โ”‚ -โ”‚ โ–ผ โ”‚ -โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ -โ”‚ โ”‚ Backend Service Proxies โ”‚ โ”‚ -โ”‚ โ”‚ โ”‚ โ”‚ -โ”‚ โ”‚ Trading Service โ”€โ”€โ†’ backend_request_duration_ms โ”‚ โ”‚ -โ”‚ โ”‚ Backtesting Service โ”€โ”€โ†’ circuit_breaker_state โ”‚ โ”‚ -โ”‚ โ”‚ ML Training Service โ”€โ”€โ†’ health_status โ”‚ โ”‚ -โ”‚ โ”‚ โ”‚ โ”‚ -โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ -โ”‚ โ”‚ โ”‚ -โ”‚ โ–ผ โ”‚ -โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ -โ”‚ โ”‚ Configuration Hot-Reload (PostgreSQL) โ”‚ โ”‚ -โ”‚ โ”‚ โ”‚ โ”‚ -โ”‚ โ”‚ NOTIFY Listener โ”€โ”€โ†’ notify_events_total โ”‚ โ”‚ -โ”‚ โ”‚ Config Cache โ”€โ”€โ†’ config_cache_hits โ”‚ โ”‚ -โ”‚ โ”‚ Reload Latency โ”€โ”€โ†’ config_reload_duration_ms โ”‚ โ”‚ -โ”‚ โ”‚ โ”‚ โ”‚ -โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ -โ”‚ โ”‚ โ”‚ -โ”‚ โ–ผ โ”‚ -โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ -โ”‚ โ”‚ Prometheus Metrics Exporter (:9090) โ”‚ โ”‚ -โ”‚ โ”‚ โ”‚ โ”‚ -โ”‚ โ”‚ HTTP Endpoint: /metrics โ”‚ โ”‚ -โ”‚ โ”‚ Format: Prometheus Text โ”‚ โ”‚ -โ”‚ โ”‚ Total Metrics: 80+ โ”‚ โ”‚ -โ”‚ โ”‚ โ”‚ โ”‚ -โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ -โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ - โ”‚ - โ”‚ Scrape every 5s - โ–ผ -โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” -โ”‚ Prometheus Server (:9099) โ”‚ -โ”‚ โ”‚ -โ”‚ โ€ข Time-series database (30 day retention) โ”‚ -โ”‚ โ€ข PromQL query engine โ”‚ -โ”‚ โ€ข Alert rule evaluation โ”‚ -โ”‚ โ”‚ -โ”‚ Alert Rules: โ”‚ -โ”‚ โ”œโ”€ AuthLatencySLAViolation (>10ฮผs) โ”‚ -โ”‚ โ”œโ”€ CircuitBreakerOpen โ”‚ -โ”‚ โ”œโ”€ BackendServiceUnhealthy โ”‚ -โ”‚ โ”œโ”€ NotifyListenerDisconnected โ”‚ -โ”‚ โ””โ”€ HighAuthFailureRate (>10%) โ”‚ -โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ - โ”‚ - โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” - โ”‚ โ”‚ - โ–ผ โ–ผ -โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” -โ”‚ Grafana (:3000) โ”‚ โ”‚ AlertManager (:9093) โ”‚ -โ”‚ โ”‚ โ”‚ โ”‚ -โ”‚ Dashboards: โ”‚ โ”‚ Alert Routing: โ”‚ -โ”‚ โ€ข Auth Overview โ”‚ โ”‚ โ€ข Slack (#foxhunt-*) โ”‚ -โ”‚ โ€ข Layer Latencies โ”‚ โ”‚ โ€ข PagerDuty (critical) โ”‚ -โ”‚ โ€ข Error Analysis โ”‚ โ”‚ โ€ข Email (warnings) โ”‚ -โ”‚ โ€ข Backend Services โ”‚ โ”‚ โ”‚ -โ”‚ โ€ข Config Hot-Reload โ”‚ โ”‚ Inhibition Rules: โ”‚ -โ”‚ โ€ข Rate Limiting โ”‚ โ”‚ โ€ข Circuit breaker open โ”‚ -โ”‚ โ”‚ โ”‚ โ€ข Backend unhealthy โ”‚ -โ”‚ Panels: 19 total โ”‚ โ”‚ โ”‚ -โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ -``` - -## Metrics Flow - -### Authentication Path - -``` -Request arrives - โ”‚ - โ”œโ”€ Start timer (Instant::now()) - โ”‚ - โ”œโ”€ JWT Extraction - โ”‚ โ””โ”€ metrics.auth.jwt_extraction_duration_us.observe(elapsed_us) - โ”‚ - โ”œโ”€ Revocation Check (Redis) - โ”‚ โ”œโ”€ metrics.auth.revocation_check_duration_us.observe(elapsed_us) - โ”‚ โ””โ”€ if revoked: metrics.auth.auth_errors_revoked_jwt.inc() - โ”‚ - โ”œโ”€ JWT Validation - โ”‚ โ”œโ”€ metrics.auth.jwt_validation_duration_us.observe(elapsed_us) - โ”‚ โ””โ”€ if invalid: metrics.auth.auth_errors_signature_failed.inc() - โ”‚ - โ”œโ”€ RBAC Permission Check - โ”‚ โ”œโ”€ metrics.auth.rbac_check_duration_us.observe(elapsed_us) - โ”‚ โ””โ”€ if denied: metrics.auth.auth_errors_permission_denied.inc() - โ”‚ - โ”œโ”€ Rate Limiting - โ”‚ โ”œโ”€ metrics.auth.rate_limit_check_duration_us.observe(elapsed_us) - โ”‚ โ””โ”€ if limited: metrics.auth.record_rate_limit(user_id) - โ”‚ - โ””โ”€ Total Auth - โ”œโ”€ metrics.auth.auth_total_duration_us.observe(total_elapsed_us) - โ”œโ”€ if total_elapsed_us < 10.0: metrics.auth.auth_sla_met.inc() - โ””โ”€ else: metrics.auth.auth_sla_exceeded.inc() -``` - -### Backend Proxy Path - -``` -Request to Backend Service - โ”‚ - โ”œโ”€ Start timer - โ”‚ - โ”œโ”€ Circuit Breaker Check - โ”‚ โ””โ”€ if open: metrics.proxy.circuit_breaker_state{service="trading"} = 2 - โ”‚ - โ”œโ”€ Execute Request - โ”‚ โ”œโ”€ On Success: - โ”‚ โ”‚ โ”œโ”€ metrics.proxy.backend_requests_total{service="trading"}.inc() - โ”‚ โ”‚ โ”œโ”€ metrics.proxy.backend_requests_success{service="trading"}.inc() - โ”‚ โ”‚ โ””โ”€ metrics.proxy.backend_request_duration_ms{service, method}.observe(elapsed_ms) - โ”‚ โ”‚ - โ”‚ โ””โ”€ On Failure: - โ”‚ โ”œโ”€ metrics.proxy.backend_requests_failure{service, error_type}.inc() - โ”‚ โ”œโ”€ if consecutive_failures > threshold: - โ”‚ โ”‚ โ””โ”€ metrics.proxy.record_circuit_breaker_trip(service) - โ”‚ โ””โ”€ metrics.proxy.update_health_status(service, false) - โ”‚ - โ””โ”€ Health Check (periodic) - โ”œโ”€ metrics.proxy.health_check_duration_ms.observe(elapsed_ms) - โ””โ”€ metrics.proxy.health_status{service} = 0 or 1 -``` - -### Configuration Hot-Reload Path - -``` -PostgreSQL NOTIFY Event - โ”‚ - โ”œโ”€ metrics.config.notify_events_total.inc() - โ”‚ - โ”œโ”€ Start timer - โ”‚ - โ”œโ”€ Fetch Configuration - โ”‚ โ”œโ”€ metrics.config.config_fetch_duration_ms.observe(elapsed_ms) - โ”‚ โ””โ”€ Check cache: - โ”‚ โ”œโ”€ if hit: metrics.config.config_cache_hits.inc() - โ”‚ โ””โ”€ if miss: metrics.config.config_cache_misses.inc() - โ”‚ - โ”œโ”€ Validate Configuration - โ”‚ โ”œโ”€ if valid: metrics.config.config_validation_success.inc() - โ”‚ โ””โ”€ if invalid: metrics.config.config_validation_failure.inc() - โ”‚ - โ”œโ”€ Reload Configuration - โ”‚ โ”œโ”€ metrics.config.record_config_reload(type, elapsed_ms) - โ”‚ โ””โ”€ Update type-specific counter: - โ”‚ โ”œโ”€ "auth" โ†’ metrics.config.config_updates_auth.inc() - โ”‚ โ”œโ”€ "routing" โ†’ metrics.config.config_updates_routing.inc() - โ”‚ โ””โ”€ "backend" โ†’ metrics.config.config_updates_backend.inc() - โ”‚ - โ””โ”€ Listener Health - โ”œโ”€ metrics.config.notify_listener_connected = 1 - โ””โ”€ if reconnect: metrics.config.notify_listener_reconnections.inc() -``` - -## Grafana Dashboard Layout - -``` -โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” -โ”‚ API Gateway - Authentication & Performance Dashboard โ”‚ -โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค -โ”‚ โ”‚ -โ”‚ [Authentication Overview] โ”‚ -โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ -โ”‚ โ”‚ Auth Requests/s โ”‚ โ”‚ Success โ”‚ โ”‚ SLA Compliance โ”‚ โ”‚ -โ”‚ โ”‚ Total: 1000 โ”‚ โ”‚ Rate: 99%โ”‚ โ”‚ <10ฮผs: 98% โ”‚ โ”‚ -โ”‚ โ”‚ Success: 990 โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ -โ”‚ โ”‚ Failure: 10 โ”‚ โ”‚ GREEN โ”‚ โ”‚ GREEN โ”‚ โ”‚ -โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ -โ”‚ โ”‚ -โ”‚ [Authentication Layer Latencies (ฮผs)] โ”‚ -โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ -โ”‚ โ”‚ โ”‚ โ”‚ -โ”‚ โ”‚ 10ฮผs โ”ค โ•ญโ”€ Total p99 โ”‚ โ”‚ -โ”‚ โ”‚ โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚ โ”‚ -โ”‚ โ”‚ 5ฮผs โ”œ โ•ญโ”€ RBAC โ”‚ โ”‚ โ”‚ -โ”‚ โ”‚ โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚ โ”‚ โ”‚ -โ”‚ โ”‚ 1ฮผs โ”œ โ•ญโ”€ JWT Validation โ”‚ โ”‚ โ”‚ โ”‚ -โ”‚ โ”‚ โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚ โ”‚ โ”‚ โ”‚ -โ”‚ โ”‚ 0.5ฮผs โ”œ JWT Extraction โ”‚ โ”‚ โ”‚ โ”‚ -โ”‚ โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”‚ โ”‚ -โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ -โ”‚ โ”‚ -โ”‚ [Error Analysis] [Cache Performance] โ”‚ -โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ -โ”‚ โ”‚ Errors by Type โ”‚ โ”‚ JWT Cache: 99.5% โ”‚ โ”‚ -โ”‚ โ”‚ - Expired: 5/s โ”‚ โ”‚ RBAC Cache: 98.2% โ”‚ โ”‚ -โ”‚ โ”‚ - Revoked: 3/s โ”‚ โ”‚ Config Cache: 95.1% โ”‚ โ”‚ -โ”‚ โ”‚ - Invalid: 2/s โ”‚ โ”‚ โ”‚ โ”‚ -โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ -โ”‚ โ”‚ -โ”‚ [Backend Services] โ”‚ -โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”โ”‚ -โ”‚ โ”‚ Backend Latency (p99 ms) โ”‚โ”‚ -โ”‚ โ”‚ Trading: 15ms โ–ˆโ–ˆโ–ˆโ–ˆโ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘ โ”‚โ”‚ -โ”‚ โ”‚ Backtesting: 250ms โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–‘โ–‘โ–‘ โ”‚โ”‚ -โ”‚ โ”‚ ML Training: 5000msโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ โ”‚โ”‚ -โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜โ”‚ -โ”‚ โ”‚ -โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ -โ”‚ โ”‚Circuit Breaker โ”‚ โ”‚ Health Status โ”‚ โ”‚ -โ”‚ โ”‚ โ”‚ โ”‚ Service Status Last Check โ”‚ โ”‚ -โ”‚ โ”‚Trading: CLOSED โ”‚ โ”‚ Trading โœ… UP 2s ago โ”‚ โ”‚ -โ”‚ โ”‚Backtest: CLOSED โ”‚ โ”‚ Backtesting โœ… UP 1s ago โ”‚ โ”‚ -โ”‚ โ”‚ML Train: CLOSED โ”‚ โ”‚ ML Training โœ… UP 3s ago โ”‚ โ”‚ -โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ -โ”‚ โ”‚ -โ”‚ [Configuration & Hot-Reload] โ”‚ -โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ -โ”‚ โ”‚ Config Reloads โ”‚ โ”‚ Reload Latency p95 โ”‚ โ”‚ NOTIFY โ”‚ โ”‚ -โ”‚ โ”‚ Auth: 5 โ”‚ โ”‚ 25ms โ”‚ โ”‚ Listener โ”‚ โ”‚ -โ”‚ โ”‚ Routing: 3 โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ -โ”‚ โ”‚ Backend: 2 โ”‚ โ”‚ โ”‚ โ”‚ CONNECTED โ”‚ โ”‚ -โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ -โ”‚ โ”‚ -โ”‚ [Rate Limiting] โ”‚ -โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ -โ”‚ โ”‚ Top 10 Rate-Limited Users โ”‚ โ”‚ -โ”‚ โ”‚ user_123: 50/s โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ โ”‚ โ”‚ -โ”‚ โ”‚ user_456: 30/s โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ โ”‚ โ”‚ -โ”‚ โ”‚ user_789: 20/s โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ โ”‚ โ”‚ -โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ -โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ -``` - -## Alert Flow - -``` -Metric Exceeds Threshold - โ”‚ - โ–ผ -Prometheus Alert Rule Triggered - โ”‚ - โ”œโ”€ Evaluates condition (e.g., auth_total_duration_us p99 > 10) - โ””โ”€ Fires alert if condition true for duration - โ”‚ - โ–ผ -AlertManager Receives Alert - โ”‚ - โ”œโ”€ Apply inhibition rules - โ”‚ โ””โ”€ Suppress redundant alerts - โ”‚ - โ”œโ”€ Route based on labels - โ”‚ โ”œโ”€ severity: critical โ†’ critical-alerts receiver - โ”‚ โ”œโ”€ severity: warning โ†’ warning-alerts receiver - โ”‚ โ””โ”€ component: auth โ†’ auth-alerts receiver - โ”‚ - โ””โ”€ Send notifications - โ”œโ”€ Slack: #foxhunt-critical - โ”œโ”€ PagerDuty: On-call engineer - โ””โ”€ Email: team@foxhunt.com -``` - -## Performance Impact - -``` -Request Processing Time Breakdown: - -โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” -โ”‚ Total Request Processing: 10,500 ns โ”‚ -โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค -โ”‚ โ”‚ -โ”‚ Authentication: 10,000 ns (95.2%) โ”‚ -โ”‚ โ”œโ”€ JWT Extraction: 500 ns โ”‚ -โ”‚ โ”œโ”€ Revocation Check: 300 ns โ”‚ -โ”‚ โ”œโ”€ JWT Validation: 800 ns โ”‚ -โ”‚ โ”œโ”€ RBAC Check: 100 ns โ”‚ -โ”‚ โ”œโ”€ Rate Limit: 50 ns โ”‚ -โ”‚ โ””โ”€ Audit Logging: 8,250 ns โ”‚ -โ”‚ โ”‚ -โ”‚ Metrics Recording: 500 ns (4.8%) โ”‚ -โ”‚ โ”œโ”€ Counter increments: 200 ns โ”‚ -โ”‚ โ”œโ”€ Histogram observations: 250 ns โ”‚ -โ”‚ โ””โ”€ Label lookups: 50 ns โ”‚ -โ”‚ โ”‚ -โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ - -Metrics Overhead: 500 ns / 10,500 ns = 4.8% -SLA Impact: 0.5 ฮผs / 10 ฮผs = 5% of SLA budget -``` - -## Data Retention - -``` -Prometheus Storage: - -โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” -โ”‚ Metric Type โ”‚ Series โ”‚ Size/Day โ”‚ -โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค -โ”‚ Counters โ”‚ 45 โ”‚ 5 MB โ”‚ -โ”‚ Histograms โ”‚ 12 โ”‚ 15 MB โ”‚ -โ”‚ Gauges โ”‚ 23 โ”‚ 3 MB โ”‚ -โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค -โ”‚ Total โ”‚ 80 โ”‚ 23 MB/day โ”‚ -โ”‚ โ”‚ -โ”‚ 30-day retention: ~700 MB โ”‚ -โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ -``` - -## Quick Reference - -### Key Metrics to Monitor - -| Priority | Metric | Target | Alert | -|----------|--------|--------|-------| -| P0 | `auth_total_duration_microseconds` (p99) | <10ฮผs | >10ฮผs | -| P0 | `health_status{service}` | 1 | 0 | -| P0 | `circuit_breaker_state{service}` | 0 | 2 | -| P1 | `backend_request_duration_milliseconds` (p99) | <50ms | >100ms | -| P1 | `auth_requests_success / total` | >99% | <95% | -| P2 | `jwt_cache_hits / (hits + misses)` | >99% | <95% | -| P2 | `notify_listener_connected` | 1 | 0 | - -### PromQL Queries - -**Auth Success Rate:** -```promql -100 * rate(api_gateway_auth_requests_success[5m]) - / rate(api_gateway_auth_requests_total[5m]) -``` - -**Auth Latency p99:** -```promql -histogram_quantile(0.99, - rate(api_gateway_auth_total_duration_microseconds_bucket[1m])) -``` - -**Cache Hit Rate:** -```promql -100 * rate(api_gateway_jwt_cache_hits[1m]) - / (rate(api_gateway_jwt_cache_hits[1m]) - + rate(api_gateway_jwt_cache_misses[1m])) -``` - -**Backend Availability:** -```promql -avg_over_time(api_gateway_health_status{service="trading"}[5m]) -``` - ---- - -*Architecture document for Wave 71 Agent 9 - Metrics Implementation* diff --git a/services/api_gateway/METRICS_DEPLOYMENT.md b/services/api_gateway/METRICS_DEPLOYMENT.md deleted file mode 100644 index 649c646b4..000000000 --- a/services/api_gateway/METRICS_DEPLOYMENT.md +++ /dev/null @@ -1,363 +0,0 @@ -# API Gateway Metrics - Deployment Guide - -## Quick Start - -### 1. Run the Monitoring Stack - -```bash -cd monitoring -docker-compose up -d -``` - -This starts: -- **Prometheus** on `http://localhost:9099` (metrics collection) -- **Grafana** on `http://localhost:3000` (visualization) -- **AlertManager** on `http://localhost:9093` (alert routing) -- **PostgreSQL Exporter** on `http://localhost:9187` -- **Redis Exporter** on `http://localhost:9121` -- **Node Exporter** on `http://localhost:9100` - -### 2. Access Grafana Dashboard - -1. Open `http://localhost:3000` -2. Login: `admin` / `foxhunt2025` -3. Navigate to **Dashboards** โ†’ **API Gateway - Authentication & Performance** - -### 3. Run Example to Generate Metrics - -```bash -cd services/api_gateway -cargo run --example metrics_example -``` - -This will: -- Generate 100 successful auth events -- Record 4 auth failures -- Simulate 50 trading service requests -- Simulate 30 backtesting requests -- Update health status for all backends -- Expose metrics at `http://localhost:9090/metrics` - -### 4. View Metrics in Prometheus - -Open `http://localhost:9099/graph` and try these queries: - -**Authentication Success Rate:** -```promql -100 * rate(api_gateway_auth_requests_success[1m]) / rate(api_gateway_auth_requests_total[1m]) -``` - -**Auth Latency p99:** -```promql -histogram_quantile(0.99, rate(api_gateway_auth_total_duration_microseconds_bucket[1m])) -``` - -**Backend Request Rate by Service:** -```promql -rate(api_gateway_backend_requests_total[1m]) -``` - -**Cache Hit Rate:** -```promql -100 * rate(api_gateway_jwt_cache_hits[1m]) / (rate(api_gateway_jwt_cache_hits[1m]) + rate(api_gateway_jwt_cache_misses[1m])) -``` - -## Production Integration - -### API Gateway Service - -```rust -use api_gateway::metrics::{GatewayMetrics, metrics_router}; -use axum::Server; - -#[tokio::main] -async fn main() -> Result<()> { - // Initialize metrics - let metrics = GatewayMetrics::new()?; - - // Start Prometheus exporter on separate port - let metrics_addr = "0.0.0.0:9090".parse()?; - let metrics_router = metrics_router(metrics.registry()); - - tokio::spawn(async move { - Server::bind(&metrics_addr) - .serve(metrics_router.into_make_service()) - .await - .expect("Metrics server failed"); - }); - - // Create auth interceptor with metrics - let auth_interceptor = AuthInterceptor::new( - jwt_service, - revocation_service, - authz_service, - rate_limiter, - audit_logger, - ); - - // Instrument auth interceptor to record metrics - let instrumented_auth = InstrumentedAuthInterceptor::new( - auth_interceptor, - metrics.auth.clone(), - ); - - // Create gRPC server with metrics - let server = Server::builder() - .layer(instrumented_auth) - .add_service(trading_proxy) - .add_service(backtesting_proxy) - .add_service(ml_training_proxy) - .serve(addr) - .await?; - - Ok(()) -} -``` - -### Docker Deployment - -Add metrics port to `docker-compose.yml`: - -```yaml -services: - api-gateway: - image: foxhunt/api-gateway:latest - ports: - - "50051:50051" # gRPC - - "9090:9090" # Prometheus metrics - environment: - - GATEWAY_BIND_ADDR=0.0.0.0:50051 - - METRICS_BIND_ADDR=0.0.0.0:9090 - - JWT_SECRET_FILE=/run/secrets/jwt_secret - - REDIS_URL=redis://redis:6379 - networks: - - foxhunt-monitoring -``` - -Update Prometheus to scrape API Gateway: - -```yaml -# monitoring/prometheus/prometheus.yml -scrape_configs: - - job_name: 'api_gateway' - static_configs: - - targets: ['api-gateway:9090'] -``` - -## Metrics Overview - -### Key Metrics to Monitor - -| Metric | Target | Alert Threshold | Description | -|--------|--------|-----------------|-------------| -| `api_gateway_auth_total_duration_microseconds` (p99) | <10ฮผs | >10ฮผs | Total auth latency SLA | -| `api_gateway_auth_requests_success` / `total` | >99% | <95% | Auth success rate | -| `api_gateway_circuit_breaker_state` | 0 (closed) | 2 (open) | Backend health | -| `api_gateway_health_status` | 1 (healthy) | 0 (unhealthy) | Service availability | -| `api_gateway_notify_listener_connected` | 1 | 0 | Config hot-reload | - -### Performance Targets - -**Authentication (HFT Requirements):** -- JWT extraction: <0.5ฮผs -- JWT validation: <1ฮผs -- Revocation check: <500ns -- RBAC check: <100ns -- Rate limit check: <50ns -- **Total auth: <10ฮผs (p99)** - -**Backend Proxy:** -- Trading Service: <50ms (p99) -- Backtesting Service: <500ms (p99) -- ML Training Service: <5000ms (p99) - -**Configuration:** -- Config reload: <50ms (p95) -- Config fetch: <25ms (p95) - -**Cache Performance:** -- JWT cache hit rate: >99% -- RBAC cache hit rate: >95% -- Config cache hit rate: >90% - -## Alerting - -### Critical Alerts - -1. **AuthLatencySLAViolation**: p99 auth latency >10ฮผs for 1m -2. **CircuitBreakerOpen**: Backend circuit breaker opened -3. **BackendServiceUnhealthy**: Health checks failing for 2m -4. **NotifyListenerDisconnected**: Hot-reload capability lost -5. **RedisConnectionFailure**: JWT revocation unavailable - -### Warning Alerts - -1. **HighAuthFailureRate**: Auth failure rate >10% for 2m -2. **HighBackendLatency**: Backend p99 >100ms for 3m -3. **ConnectionPoolExhaustion**: Pool utilization >90% for 5m -4. **LowCacheHitRate**: RBAC cache hit rate <90% for 5m -5. **ExcessiveRateLimiting**: Rate limit rejections >10/s for 5m - -## Grafana Dashboard Panels - -### Row 1: Authentication Overview -- **Auth Requests**: Total/Success/Failure rate -- **Auth Success Rate**: Percentage gauge with thresholds -- **Auth SLA Compliance**: <10ฮผs target compliance - -### Row 2: Layer Latency Breakdown -- **JWT Extraction p99**: Sub-microsecond target -- **JWT Validation p99**: <1ฮผs target -- **Revocation Check p99**: <500ns target -- **RBAC Check p99**: <100ns target -- **Rate Limit Check p99**: <50ns target -- **Total Auth p99**: <10ฮผs SLA - -### Row 3: Error Analysis -- **Errors by Type**: Missing JWT, expired, revoked, etc. -- **Cache Performance**: JWT and RBAC cache hit rates - -### Row 4: Backend Services -- **Request Latency**: p99 by service -- **Circuit Breaker States**: Open/closed status -- **Health Status**: Table view of service health -- **Connection Pool**: Utilization percentage - -### Row 5: Configuration -- **Reload Events**: By type (auth, routing, rate_limit, backend) -- **Hot-Reload Latency**: p95 reload time -- **NOTIFY Listener**: Connection status indicator - -### Row 6: Rate Limiting -- **Top 10 Rate-Limited Users**: Highest rejection rates -- **Active Entries**: Total users being tracked - -## Troubleshooting - -### Metrics Not Appearing in Prometheus - -1. Check API Gateway metrics endpoint: - ```bash - curl http://localhost:9090/metrics | grep api_gateway - ``` - -2. Verify Prometheus target: - ```bash - curl http://localhost:9099/api/v1/targets | jq '.data.activeTargets[] | select(.job=="api_gateway")' - ``` - -3. Check Prometheus logs: - ```bash - docker logs foxhunt-prometheus - ``` - -### Grafana Dashboard Not Loading - -1. Verify Prometheus data source: - - Grafana โ†’ Configuration โ†’ Data Sources - - URL should be `http://prometheus:9090` - -2. Import dashboard manually: - - Grafana โ†’ Dashboards โ†’ Import - - Upload `monitoring/grafana/api_gateway_dashboard.json` - -### High Metrics Cardinality - -If Prometheus memory usage is high: - -```bash -# Check metric cardinality -curl http://localhost:9099/api/v1/label/__name__/values | jq '. | length' - -# Identify high-cardinality metrics -curl 'http://localhost:9099/api/v1/query?query=count({__name__=~".+"}) by (__name__)' | jq '.data.result | sort_by(.value[1] | tonumber) | reverse | .[0:10]' -``` - -**Solutions:** -- Reduce `user_id` label cardinality in per-user metrics -- Use recording rules for aggregated metrics -- Decrease retention period in Prometheus config - -### Alerts Not Firing - -1. Verify alert rules syntax: - ```bash - docker exec foxhunt-prometheus promtool check rules /etc/prometheus/alerts/api_gateway_alerts.yml - ``` - -2. Check active alerts: - ```bash - curl http://localhost:9099/api/v1/alerts | jq '.data.alerts' - ``` - -3. Verify AlertManager: - ```bash - curl http://localhost:9093/api/v2/alerts | jq - ``` - -## Performance Impact - -### Metrics Overhead - -- Counter increment: **~50ns** -- Histogram observation: **~200ns** -- Label lookup: **~10ns** (cached) - -**Total overhead per authenticated request: <500ns (<0.005% of 10ฮผs SLA)** - -### Memory Usage - -- Prometheus TSDB: ~50MB per million data points -- Grafana: ~200MB base + 10MB per dashboard -- API Gateway metrics: ~5MB per 100k requests - -### Network Bandwidth - -- Metrics scrape: ~50KB per scrape -- Scrape interval: 5s -- Bandwidth: ~10KB/s per target - -## Production Checklist - -- [ ] Prometheus deployed with persistent storage -- [ ] Grafana deployed with API Gateway dashboard -- [ ] AlertManager configured with notification channels -- [ ] PostgreSQL exporter connected to config database -- [ ] Redis exporter connected to revocation cache -- [ ] API Gateway metrics endpoint exposed on :9090 -- [ ] Firewall rules allow Prometheus scraping -- [ ] HTTPS/TLS configured for Grafana -- [ ] Alert routing tested (Slack/PagerDuty) -- [ ] Metric retention policy configured (30 days default) -- [ ] Backup configured for Prometheus/Grafana data -- [ ] Monitoring stack monitored (meta-monitoring) - -## Next Steps - -1. **Configure Alerting**: - - Update Slack webhook in `alertmanager.yml` - - Configure PagerDuty for critical alerts - - Test alert routing - -2. **Customize Dashboard**: - - Add panels for business metrics - - Configure alerts for your SLAs - - Set up user-specific views - -3. **Integrate with CI/CD**: - - Add metrics checks to deployment pipeline - - Canary deployments based on error rates - - Automated rollback on SLA violations - -4. **Advanced Features**: - - Recording rules for complex queries - - Federation for multi-cluster setup - - Long-term storage (Thanos/Cortex) - -## Support - -For issues or questions: -- Check logs: `docker logs foxhunt-prometheus` -- Review Grafana docs: https://grafana.com/docs/ -- Prometheus docs: https://prometheus.io/docs/ -- Internal docs: `services/api_gateway/src/metrics/README.md` diff --git a/services/api_gateway/ML_TRAINING_PROXY_INTEGRATION.md b/services/api_gateway/ML_TRAINING_PROXY_INTEGRATION.md deleted file mode 100644 index 14266e48c..000000000 --- a/services/api_gateway/ML_TRAINING_PROXY_INTEGRATION.md +++ /dev/null @@ -1,405 +0,0 @@ -# ML Training Service Proxy Integration - -## Overview - -The ML Training Service Proxy provides zero-copy gRPC forwarding for the ML Training Service with: -- **Routing overhead**: <10ฮผs target -- **Connection pooling**: Managed by `tonic::transport::Channel` -- **Circuit breaker**: Automatic failure detection and recovery -- **Streaming support**: Efficient training metrics streaming -- **Health checking**: Backend service health monitoring - -## Architecture - -``` -Client โ†’ API Gateway (Proxy) โ†’ ML Training Service (Backend) - โ†“ - Circuit Breaker (5 failures / 30s reset) - Connection Pool (HTTP/2) - Zero-copy forwarding -``` - -## Files Created - -### 1. `src/grpc/ml_training_proxy.rs` -**Purpose**: Zero-copy gRPC proxy implementation - -**Key Features**: -- Implements `MlTrainingService` trait with all 7 RPC methods -- Zero-copy request/response forwarding -- Efficient server streaming for `SubscribeToTrainingStatus` -- UUID-based request tracing -- Comprehensive error logging - -**Performance**: -- Client cloning: O(1) (Arc increment) -- Request forwarding: Direct message passing (no deserialization) -- Stream forwarding: Zero-copy stream passthrough - -### 2. `src/grpc/server.rs` -**Purpose**: Backend client setup with circuit breaker - -**Key Components**: - -```rust -pub struct MlTrainingBackendConfig { - pub address: String, // "http://ml-training-service:50053" - pub connect_timeout_ms: u64, // Default: 5000ms - pub request_timeout_ms: u64, // Default: 30000ms - pub circuit_breaker_failures: u64, // Default: 5 failures - pub circuit_breaker_reset_secs: u64, // Default: 30s -} -``` - -**Functions**: -- `setup_ml_training_client()`: Creates client with circuit breaker -- `setup_ml_training_proxy()`: Creates ready-to-serve proxy - -### 3. `build.rs` -**Purpose**: Compile ML Training Service protobuf definitions - -**Configuration**: -- Builds both client and server code (for proxying) -- Adds serde serialization support -- Compiles from `../ml_training_service/proto/ml_training.proto` - -### 4. `src/grpc/mod.rs` -**Purpose**: Module exports - -```rust -pub use ml_training_proxy::MlTrainingProxy; -pub use server::{ - MlTrainingBackendConfig, - setup_ml_training_client, - setup_ml_training_proxy -}; -``` - -## Integration Example - -### Basic Setup - -```rust -use api_gateway::grpc::{MlTrainingBackendConfig, setup_ml_training_proxy}; -use tonic::transport::Server; - -#[tokio::main] -async fn main() -> Result<(), Box> { - // Configure ML Training Service backend - let ml_config = MlTrainingBackendConfig { - address: "http://ml-training-service:50053".to_string(), - connect_timeout_ms: 5000, - request_timeout_ms: 30000, - circuit_breaker_failures: 5, - circuit_breaker_reset_secs: 30, - }; - - // Setup proxy with circuit breaker - let ml_training_proxy = setup_ml_training_proxy(ml_config).await?; - - // Convert to tonic server - let ml_training_service = ml_training_proxy.into_server(); - - // Start gRPC server - let addr = "0.0.0.0:50051".parse()?; - Server::builder() - .add_service(ml_training_service) - .serve(addr) - .await?; - - Ok(()) -} -``` - -### With Health Checking - -```rust -use tonic_health::server::HealthReporter; - -#[tokio::main] -async fn main() -> Result<(), Box> { - // Setup ML training proxy - let ml_training_proxy = setup_ml_training_proxy( - MlTrainingBackendConfig::default() - ).await?; - - // Setup health reporter - let mut health_reporter = HealthReporter::new(); - health_reporter.set_serving::>().await; - - // Create services - let ml_training_service = ml_training_proxy.into_server(); - let health_service = health_reporter.into_service(); - - // Start server with health checking - Server::builder() - .add_service(ml_training_service) - .add_service(health_service) - .serve("0.0.0.0:50051".parse()?) - .await?; - - Ok(()) -} -``` - -### With Multiple Services - -```rust -use api_gateway::grpc::{ - TradingServiceProxy, - BacktestingServiceProxy, - MlTrainingProxy, - setup_ml_training_proxy -}; - -#[tokio::main] -async fn main() -> Result<(), Box> { - // Setup all service proxies - let trading_proxy = setup_trading_proxy(trading_config).await?; - let backtesting_proxy = setup_backtesting_proxy(backtesting_config).await?; - let ml_training_proxy = setup_ml_training_proxy(ml_training_config).await?; - - // Start unified API Gateway - Server::builder() - .add_service(trading_proxy.into_server()) - .add_service(backtesting_proxy.into_server()) - .add_service(ml_training_proxy.into_server()) - .serve("0.0.0.0:50051".parse()?) - .await?; - - Ok(()) -} -``` - -## RPC Methods Supported - -### 1. `StartTraining` (Unary) -```rust -rpc StartTraining(StartTrainingRequest) returns (StartTrainingResponse) -``` -- **Performance**: <10ฮผs routing overhead -- **Error handling**: Circuit breaker on backend failures - -### 2. `SubscribeToTrainingStatus` (Server Streaming) -```rust -rpc SubscribeToTrainingStatus(SubscribeToTrainingStatusRequest) - returns (stream TrainingStatusUpdate) -``` -- **Performance**: Zero-copy stream forwarding -- **No buffering**: Direct stream passthrough from backend - -### 3. `StopTraining` (Unary) -```rust -rpc StopTraining(StopTrainingRequest) returns (StopTrainingResponse) -``` - -### 4. `ListAvailableModels` (Unary) -```rust -rpc ListAvailableModels(ListAvailableModelsRequest) - returns (ListAvailableModelsResponse) -``` - -### 5. `ListTrainingJobs` (Unary) -```rust -rpc ListTrainingJobs(ListTrainingJobsRequest) - returns (ListTrainingJobsResponse) -``` - -### 6. `GetTrainingJobDetails` (Unary) -```rust -rpc GetTrainingJobDetails(GetTrainingJobDetailsRequest) - returns (GetTrainingJobDetailsResponse) -``` - -### 7. `HealthCheck` (Unary) -```rust -rpc HealthCheck(HealthCheckRequest) returns (HealthCheckResponse) -``` - -## Performance Characteristics - -### Latency Breakdown - -| Operation | Latency | Notes | -|-----------|---------|-------| -| Client clone | ~1-2ns | Arc increment | -| Request forward | 5-8ฮผs | Target: <10ฮผs | -| Stream setup | ~10ฮผs | One-time per stream | -| Stream item forward | <1ฮผs | Zero-copy passthrough | -| Circuit breaker check | <10ฮผs | Atomic operations | - -### Memory Usage - -- **Client**: ~200 bytes (Arc to Channel) -- **Proxy**: ~200 bytes (contains client) -- **Per-request overhead**: 0 bytes (zero-copy) -- **Stream overhead**: ~1KB buffer per stream - -### Connection Pooling - -- **HTTP/2 multiplexing**: Unlimited concurrent streams per connection -- **Connection reuse**: Automatic via `tonic::transport::Channel` -- **Keepalive**: 60s TCP keepalive, 30s HTTP/2 keepalive - -## Circuit Breaker Behavior - -### States - -1. **Closed** (Normal operation) - - Requests forwarded normally - - Failures counted - -2. **Open** (Backend unavailable) - - Requests fail immediately - - No backend calls - - After reset timeout โ†’ Half-Open - -3. **Half-Open** (Testing recovery) - - Single probe request allowed - - Success โ†’ Closed - - Failure โ†’ Open - -### Configuration - -```rust -MlTrainingBackendConfig { - circuit_breaker_failures: 5, // Open after 5 consecutive failures - circuit_breaker_reset_secs: 30, // Try to close after 30 seconds - ..Default::default() -} -``` - -## Error Handling - -### Backend Connection Failures -```rust -Status::unavailable("ML Training Service circuit breaker: connection refused") -``` - -### Backend Request Timeouts -```rust -Status::deadline_exceeded("Request timeout after 30000ms") -``` - -### Circuit Breaker Open -```rust -Status::unavailable("ML Training Service circuit breaker: circuit open") -``` - -## Monitoring and Logging - -All requests include: -- UUID-based request tracing -- Structured logging with `tracing` crate -- Error logging with full context -- Performance tracing via `#[instrument]` macro - -### Example Logs - -``` -INFO Proxying StartTraining request request_id=abc-123 -INFO StartTraining request forwarded successfully request_id=abc-123 - -INFO Proxying SubscribeToTrainingStatus streaming request request_id=def-456 -INFO SubscribeToTrainingStatus streaming request forwarded successfully request_id=def-456 - -ERROR Backend StartTraining failed: status: Unavailable, ... -``` - -## Testing - -### Unit Tests -```bash -cargo test -p api_gateway --lib grpc::ml_training_proxy -``` - -### Integration Tests -```bash -# Start ML Training Service backend -cargo run -p ml_training_service - -# Start API Gateway with ML Training proxy -cargo run -p api_gateway - -# Test via gRPC client -grpcurl -plaintext localhost:50051 ml_training.MLTrainingService/StartTraining -``` - -## Production Deployment - -### Environment Variables - -```bash -# ML Training Service backend address -ML_TRAINING_SERVICE_ADDR=http://ml-training-service:50053 - -# Connection timeouts -ML_TRAINING_CONNECT_TIMEOUT_MS=5000 -ML_TRAINING_REQUEST_TIMEOUT_MS=30000 - -# Circuit breaker configuration -ML_TRAINING_CIRCUIT_FAILURES=5 -ML_TRAINING_CIRCUIT_RESET_SECS=30 - -# API Gateway listen address -API_GATEWAY_ADDR=0.0.0.0:50051 -``` - -### Docker Deployment - -```yaml -services: - api-gateway: - image: foxhunt/api-gateway:latest - environment: - - ML_TRAINING_SERVICE_ADDR=http://ml-training-service:50053 - - ML_TRAINING_CONNECT_TIMEOUT_MS=5000 - - ML_TRAINING_REQUEST_TIMEOUT_MS=30000 - ports: - - "50051:50051" - depends_on: - - ml-training-service - - ml-training-service: - image: foxhunt/ml-training-service:latest - ports: - - "50053:50053" -``` - -## Benchmarks - -### Target Performance - -- Routing overhead: <10us (5-8us typical) -- Zero-copy forwarding: Implemented -- Connection pooling: Via tonic::Channel -- Circuit breaker: <10us overhead -- Streaming support: Zero-copy passthrough - -### Measurement - -```rust -use std::time::Instant; - -let start = Instant::now(); -let response = proxy.start_training(request).await?; -let latency = start.elapsed(); - -println!("Routing latency: {}ฮผs", latency.as_micros()); -``` - -## Future Enhancements - -1. **Metrics Collection**: Prometheus metrics for latency, throughput, errors -2. **Request Caching**: Cache expensive operations (ListAvailableModels) -3. **Load Balancing**: Multiple backend instances -4. **Rate Limiting**: Per-user request limits -5. **Request Validation**: Schema validation before forwarding - -## References - -- ML Training Service proto: `/services/ml_training_service/proto/ml_training.proto` -- Proxy implementation: `/services/api_gateway/src/grpc/ml_training_proxy.rs` -- Server setup: `/services/api_gateway/src/grpc/server.rs` -- Build configuration: `/services/api_gateway/build.rs` - diff --git a/services/api_gateway/RATE_LIMITER_IMPLEMENTATION.md b/services/api_gateway/RATE_LIMITER_IMPLEMENTATION.md deleted file mode 100644 index 634fd7805..000000000 --- a/services/api_gateway/RATE_LIMITER_IMPLEMENTATION.md +++ /dev/null @@ -1,357 +0,0 @@ -# Rate Limiter Implementation - -> **Note:** HTTP-level rate limiting for the web dashboard is now handled by `web-gateway` (3 tiers: auth 10/min, trading 200/min, compute 30/min). This document describes the gRPC-level rate limiting in api_gateway. - -## Overview - -High-performance token bucket rate limiting system with Redis backend and in-memory caching optimized for HFT requirements. - -## Architecture - -### Token Bucket Algorithm - -The rate limiter uses the token bucket algorithm which provides: -- **Smooth rate limiting** - requests consume tokens from a bucket -- **Burst handling** - bucket capacity allows short bursts up to limit -- **Automatic refill** - tokens refill at a constant rate over time - -### Two-Tier Architecture - -``` -โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” -โ”‚ Rate Limiter โ”‚ -โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค -โ”‚ โ”‚ -โ”‚ 1. In-Memory Cache (LRU, 10,000 entries) โ”‚ -โ”‚ โ””โ”€ HashMap โ”‚ -โ”‚ โ””โ”€ Target: <50ns per check (CACHE HIT) โ”‚ -โ”‚ โ””โ”€ TTL: 1 second โ”‚ -โ”‚ โ”‚ -โ”‚ 2. Redis Backend (Lua scripts) โ”‚ -โ”‚ โ””โ”€ Atomic token bucket operations โ”‚ -โ”‚ โ””โ”€ Target: <500ฮผs per check (CACHE MISS) โ”‚ -โ”‚ โ””โ”€ Distributed rate limiting across instances โ”‚ -โ”‚ โ”‚ -โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ -``` - -## Performance Characteristics - -### Measured Performance - -Based on standalone benchmarks: - -| Operation | Target | Measured | Status | -|-----------|--------|----------|--------| -| Cache Hit | <50ns | 25ns | โœ“ EXCELLENT | -| Token Bucket | N/A | 625ns | โœ“ | -| Burst Handling | N/A | 41ns/req | โœ“ | -| HFT Scenario | N/A | 27ns/check | โœ“ | - -### Performance Breakdown - -1. **In-Memory Cache Hit**: 25-41ns - - HashMap lookup: ~5ns - - Token bucket check: ~10ns - - Refill calculation: ~10ns - - Lock overhead: ~10ns - -2. **Redis Backend**: <500ฮผs (estimated) - - Network RTT: ~100-200ฮผs (same AZ) - - Lua script execution: ~50-100ฮผs - - Serialization: ~50ฮผs - - Connection pool overhead: ~50ฮผs - -## Implementation Details - -### Rate Limit Configurations - -Pre-configured limits for common endpoints: - -```rust -// Trading endpoints (high frequency) -trading.submit_order: - - Capacity: 100 tokens - - Refill rate: 100 tokens/second - - Burst size: 10 requests - -// Configuration updates (moderate frequency) -config.update: - - Capacity: 10 tokens - - Refill rate: 10 tokens/second - - Burst size: 2 requests - -// Backtesting (low frequency) -backtesting.run: - - Capacity: 5 tokens - - Refill rate: 5 tokens/60 seconds (5/min) - - Burst size: 1 request - -// Default (for unlisted endpoints) -default: - - Capacity: 50 tokens - - Refill rate: 50 tokens/second - - Burst size: 5 requests -``` - -### Redis Lua Script - -Atomic token bucket implementation in Redis: - -```lua -local key = KEYS[1] -local capacity = tonumber(ARGV[1]) -local refill_rate = tonumber(ARGV[2]) -local now = tonumber(ARGV[3]) - --- Get current bucket state -local bucket = redis.call('HGETALL', key) -local tokens = capacity -local last_refill = now - --- Parse existing state -if #bucket > 0 then - for i = 1, #bucket, 2 do - if bucket[i] == 'tokens' then - tokens = tonumber(bucket[i + 1]) - elseif bucket[i] == 'last_refill' then - last_refill = tonumber(bucket[i + 1]) - end - end -end - --- Refill tokens based on elapsed time -local elapsed = now - last_refill -tokens = math.min(capacity, tokens + (elapsed * refill_rate)) - --- Check if request is allowed -if tokens >= 1 then - tokens = tokens - 1 - redis.call('HSET', key, 'tokens', tokens, 'last_refill', now) - redis.call('EXPIRE', key, 300) -- 5 minute TTL - return 1 -else - redis.call('HSET', key, 'tokens', tokens, 'last_refill', now) - redis.call('EXPIRE', key, 300) - return 0 -end -``` - -### LRU Cache Management - -```rust -// Cache configuration -max_cache_size: 10,000 entries -cache_ttl: 1 second - -// Eviction policy -- When cache reaches 10,000 entries -- Evict oldest 10% (1,000 entries) -- Based on last_access timestamp -- Automatic on cache miss -``` - -## Integration Points - -### AuthInterceptor Integration - -```rust -// In services/api_gateway/src/auth/interceptor.rs -impl AuthInterceptor { - pub async fn intercept(&self, request: Request<()>) -> Result, Status> { - // ... JWT validation ... - - // Layer 6: Rate Limiting (<50ns) - let endpoint = extract_endpoint(&request); - if !self.rate_limiter - .check_limit(&claims.user_id, endpoint) - .await - .map_err(|_| Status::internal("Rate limit check failed"))? - { - return Err(Status::resource_exhausted("Rate limit exceeded")); - } - - // ... continue processing ... - } -} -``` - -### Configuration Loading - -Rate limit configurations are loaded from PostgreSQL: - -```sql --- Example: Update rate limit for endpoint -UPDATE rate_limit_config -SET capacity = 200, refill_rate = 200 -WHERE endpoint = 'trading.submit_order'; - --- Triggers PostgreSQL NOTIFY for hot-reload -NOTIFY config_updates, 'rate_limit_config'; -``` - -## Testing - -### Unit Tests - -```bash -# Run rate limiter unit tests -cargo test -p api_gateway rate_limiter - -# Expected output: -# test rate_limiter::tests::test_token_bucket_basic ... ok -# test rate_limiter::tests::test_token_bucket_refill ... ok -# test rate_limiter::tests::test_rate_limit_configs ... ok -``` - -### Benchmarks - -```bash -# Run performance benchmarks -rustc services/api_gateway/benches/rate_limiter_bench.rs -O && ./rate_limiter_bench - -# Expected output: -# Cache hit: 25 ns (target <50ns) โœ“ -# Token bucket: 625 ns โœ“ -# Burst handling: 41 ns โœ“ -# HFT scenario: 27 ns โœ“ -``` - -### Integration Testing - -```bash -# Start Redis for testing -docker run -d -p 6379:6379 redis:7-alpine - -# Run integration tests (when implemented) -cargo test -p api_gateway --test rate_limiter_integration -``` - -## Redis Setup - -### Development - -```bash -# Docker -docker run -d -p 6379:6379 --name foxhunt-redis redis:7-alpine - -# Or local Redis -redis-server --port 6379 -``` - -### Production - -```bash -# Environment variables -export REDIS_URL="redis://redis-cluster.internal:6379" -export RATE_LIMITER_CACHE_SIZE=10000 -export RATE_LIMITER_CACHE_TTL_SECONDS=1 -``` - -## Monitoring - -### Metrics to Track - -1. **Cache Hit Rate** - - Target: >95% hit rate - - Monitor: `cache_hits / (cache_hits + cache_misses)` - -2. **Latency Distribution** - - p50: <30ns (cache hits) - - p95: <100ns (cache hits) - - p99: <1ฮผs (Redis hits) - -3. **Rate Limit Violations** - - Track denied requests per endpoint - - Alert on unusual patterns - -### Cache Statistics - -```rust -// Get current cache statistics -let stats = rate_limiter.get_cache_stats().await; -println!("Cache size: {}/{}", stats.size, stats.max_size); -println!("Cache TTL: {} seconds", stats.ttl_seconds); -``` - -## Future Enhancements - -1. **Dynamic Configuration** - - Load rate limits from PostgreSQL config table - - Hot-reload on configuration changes via NOTIFY/LISTEN - -2. **Advanced Metrics** - - Prometheus metrics integration - - Per-endpoint rate limit statistics - - User-level quota tracking - -3. **Sliding Window Algorithm** - - Optional sliding window rate limiting - - More accurate but slightly higher overhead - -4. **Distributed Cache** - - Redis as primary cache (shared across instances) - - Local in-memory as L2 cache - - Eventual consistency acceptable for rate limiting - -## Files Created - -``` -services/api_gateway/src/routing/ -โ”œโ”€โ”€ mod.rs # Module exports -โ””โ”€โ”€ rate_limiter.rs # Token bucket implementation - -services/api_gateway/benches/ -โ””โ”€โ”€ rate_limiter_bench.rs # Performance benchmarks - -docs/ -โ””โ”€โ”€ RATE_LIMITER_IMPLEMENTATION.md # This file -``` - -## Performance Validation - -### Benchmark Results - -``` -Rate Limiter Performance Benchmarks -======================================== - -Benchmark 1: Cache Hit Performance (in-memory) -Total time: 25.167394ms -Operations: 1000000 -Time per operation: 25 ns -Target: <50ns โœ“ - -Benchmark 2: Token Bucket Refill Overhead -Total time: 62.599479ms -Operations: 100000 -Time per operation: 625 ns -(includes 10ฮผs sleeps every 100 operations) - -Benchmark 3: Burst Handling (100 requests at once) -Total time: 4.177ยตs -Allowed requests: 100/100 -Average per request: 41 ns - -Benchmark 4: HFT Scenario (10,000 requests, 100 req/sec limit) -Total time: 272.411ยตs -Allowed: 100/10000 requests -Denied: 9900 requests -Average per check: 27 ns - -======================================== -Performance Summary: - - Cache hit: 25 ns (target <50ns) โœ“ - - Token bucket: 625 ns โœ“ - - Burst handling: 41 ns โœ“ - - HFT scenario: 27 ns โœ“ -``` - -## Status - -Implementation complete. Key files: - -- `services/api_gateway/src/routing/rate_limiter.rs` - Token bucket implementation -- `services/api_gateway/benches/rate_limiter_bench.rs` - Performance benchmarks -- Unit tests included in `rate_limiter.rs` - diff --git a/services/api_gateway/REVOCATION_CACHE_USAGE.md b/services/api_gateway/REVOCATION_CACHE_USAGE.md deleted file mode 100644 index 1c4ad954f..000000000 --- a/services/api_gateway/REVOCATION_CACHE_USAGE.md +++ /dev/null @@ -1,366 +0,0 @@ -# Revocation Cache - Quick Reference - -## Overview - -The local revocation cache eliminates Redis network latency for JWT revocation checks, improving authentication performance by 19x on average. - -## Performance - -- **Cache Hit**: <10ns (DashMap lookup) -- **Cache Miss**: ~500ฮผs (Redis network) -- **Hit Rate**: 95-99% (typical production workload) -- **Throughput**: 38K req/s (up from 10K req/s) - -## Basic Usage - -### Default Configuration (Recommended) - -```rust -use api_gateway::auth::RevocationService; - -// Create service with 60s TTL (recommended) -let revocation_service = RevocationService::new("redis://localhost:6379").await?; - -// Check if token is revoked (automatic caching) -let jti = Jti::from_string("token-id-here".to_string()); -let is_revoked = revocation_service.is_revoked(&jti).await?; -``` - -### Custom TTL Configuration - -```rust -use std::time::Duration; - -// Create service with custom 30s TTL -let revocation_service = RevocationService::new_with_cache_ttl( - "redis://localhost:6379", - Duration::from_secs(30), -).await?; -``` - -## Monitoring - -### Get Cache Statistics - -```rust -use api_gateway::auth::CacheStats; - -// Get current cache stats -let stats: CacheStats = revocation_service.cache_stats(); - -println!("Cache Hit Rate: {:.2}%", stats.hit_rate); -println!("Total Hits: {}", stats.hits); -println!("Total Misses: {}", stats.misses); -println!("Total Requests: {}", stats.total); -println!("Cached Entries: {}", stats.entries); -``` - -### Example Output - -``` -Cache Hit Rate: 97.50% -Total Hits: 9750 -Total Misses: 250 -Total Requests: 10000 -Cached Entries: 856 -``` - -## Cache Management - -### Invalidate Single Token - -```rust -// Revoke token (automatically invalidates cache) -let jti = Jti::from_string("token-to-revoke".to_string()); -revocation_service.revoke_token(&jti, 3600).await?; -// Cache entry is immediately invalidated -``` - -### Clear Entire Cache - -```rust -// Emergency cache flush (use with caution) -revocation_service.clear_cache(); -``` - -### Reset Statistics - -```rust -// Reset metrics counters -revocation_service.reset_cache_stats(); -``` - -## Configuration Guide - -### TTL Selection - -| TTL | Use Case | Hit Rate | Revocation Delay | -|-----|----------|----------|------------------| -| 30s | High security | 85-95% | 30s max | -| 60s | **Recommended** | 95-99% | 60s max | -| 120s | High performance | 99%+ | 120s max | - -**Recommendation**: 60s balances performance (95-99% hit rate) with security (acceptable revocation delay). - -### Memory Planning - -| Active Sessions | Memory Usage | -|-----------------|--------------| -| 1,000 | ~64 KB | -| 10,000 | ~640 KB | -| 100,000 | ~6.4 MB | -| 1,000,000 | ~64 MB | - -**Formula**: ~64 bytes per cached token - -## Prometheus Metrics (Future) - -### Recommended Metrics - -```prometheus -# Cache hit rate -revocation_cache_hit_rate{service="api_gateway"} 97.5 - -# Total requests -revocation_cache_requests_total{service="api_gateway"} 10000 - -# Cache hits -revocation_cache_hits_total{service="api_gateway"} 9750 - -# Cache misses -revocation_cache_misses_total{service="api_gateway"} 250 - -# Cached entries -revocation_cache_entries{service="api_gateway"} 856 -``` - -### Grafana Dashboard Example - -```json -{ - "title": "Revocation Cache Performance", - "panels": [ - { - "title": "Hit Rate", - "query": "revocation_cache_hit_rate", - "type": "gauge", - "thresholds": [90, 95, 99] - }, - { - "title": "Requests/sec", - "query": "rate(revocation_cache_requests_total[1m])", - "type": "graph" - } - ] -} -``` - -## Troubleshooting - -### Low Hit Rate (<90%) - -**Symptoms**: `cache_stats().hit_rate < 90.0` - -**Possible Causes**: -- TTL too short for access patterns -- Many unique tokens (e.g., one-time tokens) -- Rapid token rotation - -**Solutions**: -1. Increase TTL: `new_with_cache_ttl(..., Duration::from_secs(120))` -2. Check token lifetime matches cache TTL -3. Monitor token access patterns - -### High Memory Usage - -**Symptoms**: Cached entries growing unbounded - -**Possible Causes**: -- TTL not expiring entries (bug) -- Extremely high session count - -**Solutions**: -1. Verify TTL is working: `cache_stats().entries` should stabilize -2. Reduce TTL to increase turnover -3. Consider LRU eviction (future enhancement) - -### Cache Invalidation Delay - -**Symptoms**: Revoked tokens still accepted for up to TTL duration - -**Expected Behavior**: This is by design (eventual consistency) - -**Mitigation**: -- Reduce TTL for high-security deployments -- Manual invalidation: `revoke_token()` immediately clears cache -- Token lifetime should be short (<1 hour) - -## Testing - -### Unit Tests - -```bash -# Run cache-specific tests -cargo test -p api_gateway --lib auth::interceptor::tests::test_cache - -# Run all auth tests -cargo test -p api_gateway --lib auth::interceptor::tests -``` - -### Benchmarks - -```bash -# Run comprehensive cache benchmarks -cargo bench -p api_gateway --bench revocation_cache_perf - -# Run specific benchmark -cargo bench -p api_gateway --bench revocation_cache_perf -- cache_hit_latency -``` - -## Performance Tips - -### Optimize for Cache Hits - -1. **Long-lived tokens**: Use access tokens with 1-hour lifetime -2. **Session persistence**: Encourage session reuse -3. **Token rotation**: Avoid frequent token refresh - -### Monitor Cache Effectiveness - -```rust -// Log cache stats periodically -tokio::spawn(async move { - loop { - tokio::time::sleep(Duration::from_secs(60)).await; - let stats = revocation_service.cache_stats(); - info!( - "Cache stats: hit_rate={:.2}%, entries={}", - stats.hit_rate, stats.entries - ); - - if stats.hit_rate < 90.0 { - warn!("Low cache hit rate: {:.2}%", stats.hit_rate); - } - } -}); -``` - -## Security Considerations - -### Revocation Propagation Delay - -- **Issue**: Revoked tokens may be accepted for up to TTL duration -- **Acceptable for HFT**: 60s delay is reasonable for financial trading -- **Mitigation**: Short token lifetimes (<1 hour) - -### Cache Poisoning - -- **Risk**: Invalid data in cache -- **Mitigation**: - - Redis is source of truth - - TTL limits exposure window - - Manual cache clear available - -### Memory Exhaustion - -- **Risk**: Unbounded cache growth -- **Mitigation**: - - TTL-based auto-expiration - - Monitoring: `cache_stats().entries` - - Alert on excessive growth - -## Production Checklist - -- [ ] Configure appropriate TTL (default 60s recommended) -- [ ] Set up monitoring (CacheStats API) -- [ ] Configure alerting (hit rate <90%, high memory) -- [ ] Test cache invalidation in staging -- [ ] Monitor memory usage in production -- [ ] Set up Prometheus/Grafana dashboards -- [ ] Document operational procedures -- [ ] Train operations team on cache management - -## API Reference - -### RevocationService Methods - -```rust -// Factory methods -pub async fn new(redis_url: &str) -> Result -pub async fn new_with_cache_ttl(redis_url: &str, cache_ttl: Duration) -> Result - -// Core operations -pub async fn is_revoked(&self, jti: &Jti) -> Result -pub async fn revoke_token(&self, jti: &Jti, ttl_seconds: u64) -> Result<()> - -// Cache management -pub fn cache_stats(&self) -> CacheStats -pub fn clear_cache(&self) -pub fn reset_cache_stats(&self) -``` - -### CacheStats Fields - -```rust -pub struct CacheStats { - pub hits: u64, // Total cache hits - pub misses: u64, // Total cache misses - pub total: u64, // Total requests (hits + misses) - pub hit_rate: f64, // Hit rate percentage (0.0-100.0) - pub entries: usize, // Current cached entries -} -``` - -## Example: Complete Integration - -```rust -use api_gateway::auth::{RevocationService, CacheStats}; -use std::time::Duration; -use tracing::info; - -#[tokio::main] -async fn main() -> anyhow::Result<()> { - // Initialize service with 60s TTL - let revocation_service = RevocationService::new("redis://localhost:6379").await?; - - // Start monitoring task - let service_clone = revocation_service.clone(); - tokio::spawn(async move { - loop { - tokio::time::sleep(Duration::from_secs(60)).await; - let stats = service_clone.cache_stats(); - info!( - "Cache: {:.2}% hit rate, {} entries, {}/{} hits/misses", - stats.hit_rate, stats.entries, stats.hits, stats.misses - ); - } - }); - - // Use in authentication flow - let jti = Jti::from_string("user-token-123".to_string()); - let is_revoked = revocation_service.is_revoked(&jti).await?; - - if is_revoked { - // Token is revoked - reject request - } else { - // Token is valid - proceed with authentication - } - - Ok(()) -} -``` - -## Further Reading - -- **Full Documentation**: `docs/WAVE74_AGENT5_REVOCATION_CACHE.md` -- **Performance Analysis**: `docs/WAVE74_AGENT5_PERFORMANCE_SUMMARY.txt` -- **Benchmarks**: `services/api_gateway/benches/revocation_cache_perf.rs` -- **Tests**: `services/api_gateway/src/auth/interceptor.rs` (lines 774-979) - -## Support - -For questions or issues: -1. Check documentation in `docs/WAVE74_AGENT5_*.md` -2. Review benchmark results -3. Consult cache statistics via `cache_stats()` -4. Check logs for cache performance warnings diff --git a/services/api_gateway/benches/README.md b/services/api_gateway/benches/README.md deleted file mode 100644 index 7caa95b57..000000000 --- a/services/api_gateway/benches/README.md +++ /dev/null @@ -1,294 +0,0 @@ -# API Gateway Benchmarks - Quick Reference - -## Quick Start - -```bash -# Run all benchmarks -cargo bench --benches - -# Run specific benchmark suite -cargo bench --bench auth_overhead -cargo bench --bench routing_latency -cargo bench --bench rate_limiting_perf -cargo bench --bench cache_performance -cargo bench --bench throughput - -# View HTML reports -open target/criterion/report/index.html -``` - -## Benchmark Suites Summary - -| File | Benchmarks | Focus Area | Target | -|------|-----------|------------|--------| -| `auth_overhead.rs` | 8 | 8-layer auth pipeline | <10ฮผs total | -| `routing_latency.rs` | 8 | End-to-end routing | <10ฮผs overhead | -| `rate_limiting_perf.rs` | 10 | Rate limiter performance | <50ns | -| `cache_performance.rs` | 10 | Cache hit/miss latency | <100ns hit | -| `throughput.rs` | 10 | Concurrent throughput | >100K req/s | - -**Total**: 46 individual benchmarks - -## Performance Targets at a Glance - -``` -Layer 1: JWT Extraction <100ns โœ“ (~45ns) -Layer 2: JWT Validation <1ฮผs โœ“ (~910ns) -Layer 3: Revocation Check <500ns โœ“ (~13ns) -Layer 4: RBAC Check <100ns โœ“ (~8ns) -Layer 5: Rate Limiting <50ns โœ“ (~3.5ns) -Layer 6: User Context <50ns โœ“ (~7ns) -Layer 7: Audit Logging async โœ“ (non-blocking) -Layer 8: Metrics Recording <20ns โœ“ (atomic) - -Total Pipeline: <10ฮผs โœ“ (~1ฮผs) -Throughput: >100K โœ“ (~145K req/s) -``` - -## Example Output - -``` -jwt_signature_validation - time: [892.34 ns 910.12 ns 935.87 ns] -Found 12 outliers among 100 measurements (12.00%) - 4 (4.00%) high mild - 8 (8.00%) high severe - -8_layer_auth_pipeline - time: [945.23 ns 978.45 ns 1.02 ฮผs] - change: [-1.2345% +0.8901% +2.3456%] - -throughput/100k_req_target - time: [7.45 ฮผs 7.63 ฮผs 7.89 ฮผs] - thrpt: [126.7K elem/s 131.1K elem/s 134.2K elem/s] -``` - -## Advanced Usage - -### Run Specific Benchmark -```bash -cargo bench --bench auth_overhead -- jwt_validation -``` - -### Baseline Comparison -```bash -# Save baseline -cargo bench --bench auth_overhead -- --save-baseline before - -# Make changes... - -# Compare -cargo bench --bench auth_overhead -- --baseline before -``` - -### Sample Size Control -```bash -# Quick run (10 samples) -cargo bench --benches -- --sample-size 10 - -# Accurate run (200 samples) -cargo bench --benches -- --sample-size 200 -``` - -### Measurement Time -```bash -# Quick measurement (1 second) -cargo bench --benches -- --measurement-time 1 - -# Long measurement (10 seconds) -cargo bench --benches -- --measurement-time 10 -``` - -### Warm-up Time -```bash -# Skip warm-up -cargo bench --benches -- --warm-up-time 0 - -# Long warm-up (5 seconds) -cargo bench --benches -- --warm-up-time 5 -``` - -## Interpreting Results - -### Time Ranges -- `[lower median upper]` - 25th, 50th, 75th percentiles -- Lower is better -- Narrow range = consistent performance - -### Change Detection -- `[-2.3% +0.5% +3.2%]` - Performance change range -- `p = 0.23 > 0.05` - Not statistically significant -- Green = improvement, Yellow = no change, Red = regression - -### Outliers -- `12 outliers (12%)` - Statistical outliers removed -- High mild/severe = extreme measurements -- Too many outliers = unstable benchmark - -### Throughput -- `[126.7K elem/s 131.1K elem/s 134.2K elem/s]` -- Higher is better -- Elements = requests processed - -## Optimization Workflow - -1. **Establish Baseline** - ```bash - cargo bench --benches -- --save-baseline main - ``` - -2. **Make Changes** - - Optimize code - - Refactor algorithms - - Change data structures - -3. **Re-run Benchmarks** - ```bash - cargo bench --benches -- --baseline main - ``` - -4. **Analyze Results** - - Green = improvement (keep) - - Red = regression (revert or investigate) - - Yellow = no change (neutral) - -5. **Iterate** - - Focus on red benchmarks - - Profile with `perf` or `flamegraph` - - Apply optimizations - -## Common Issues - -### Noisy Results -**Problem**: Large variance in measurements -**Solution**: -```bash -# Close background apps -# Set CPU governor to performance -echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor - -# Increase sample size -cargo bench -- --sample-size 200 -``` - -### Compilation Time -**Problem**: Benchmarks take too long to compile -**Solution**: -```bash -# Build in release mode first -cargo build --release --benches - -# Then run -cargo bench --benches -``` - -### Out of Memory -**Problem**: Throughput benchmarks consume too much memory -**Solution**: -```bash -# Reduce iteration count -cargo bench --bench throughput -- --sample-size 10 -``` - -## Performance Tips - -### CPU Governor -```bash -# Linux: Set to performance mode -echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor - -# macOS: Disable Turbo Boost -sudo nvram boot-args="serverperfmode=1 $(nvram boot-args 2>/dev/null | cut -f 2-)" -``` - -### CPU Pinning -```bash -# Run on specific CPU cores -taskset -c 0,1 cargo bench --benches -``` - -### Disable Frequency Scaling -```bash -# Linux -sudo cpupower frequency-set --governor performance - -# Verify -cpupower frequency-info -``` - -## CI/CD Integration - -### GitHub Actions -```yaml -- name: Run benchmarks - run: cargo bench --benches -- --output-format bencher - -- name: Store results - uses: benchmark-action/github-action-benchmark@v1 - with: - tool: 'cargo' - output-file-path: target/criterion/output.json -``` - -### GitLab CI -```yaml -benchmark: - script: - - cargo bench --benches - artifacts: - paths: - - target/criterion/ -``` - -## File Structure - -``` -benches/ -โ”œโ”€โ”€ auth_overhead.rs # 8-layer auth pipeline (8 benchmarks) -โ”œโ”€โ”€ routing_latency.rs # End-to-end routing (8 benchmarks) -โ”œโ”€โ”€ rate_limiting_perf.rs # Rate limiter (10 benchmarks) -โ”œโ”€โ”€ cache_performance.rs # Caching layers (10 benchmarks) -โ”œโ”€โ”€ throughput.rs # Concurrent requests (10 benchmarks) -โ””โ”€โ”€ README.md # This file - -Reports: -target/criterion/ -โ”œโ”€โ”€ report/ -โ”‚ โ””โ”€โ”€ index.html # Main HTML report -โ”œโ”€โ”€ auth_overhead/ -โ”‚ โ””โ”€โ”€ jwt_validation/ -โ”‚ โ”œโ”€โ”€ base/ -โ”‚ โ”‚ โ””โ”€โ”€ estimates.json -โ”‚ โ””โ”€โ”€ new/ -โ”‚ โ””โ”€โ”€ estimates.json -โ””โ”€โ”€ ... -``` - -## Key Metrics Glossary - -- **P50 (Median)**: 50% of samples are faster -- **P95**: 95% of samples are faster -- **P99**: 99% of samples are faster -- **Throughput**: Operations per second -- **Latency**: Time per operation -- **Outliers**: Measurements removed from analysis -- **Change**: Performance delta from baseline - -## Resources - -- ๐Ÿ“Š [Criterion.rs Book](https://bheisler.github.io/criterion.rs/book/) -- ๐Ÿš€ [Rust Performance Book](https://nnethercote.github.io/perf-book/) -- ๐Ÿ”ฅ [Flamegraph Profiling](https://github.com/flamegraph-rs/flamegraph) -- ๐Ÿ“ˆ [Benchmarking Best Practices](https://easyperf.net/blog/) - -## Support - -For questions or issues: -1. Check `BENCHMARKS.md` for detailed documentation -2. Review Criterion documentation -3. Profile with `cargo flamegraph` -4. Analyze assembly with `cargo asm` - ---- - -**Wave 71 Agent 4** - Performance Benchmarking Suite diff --git a/services/api_gateway/src/metrics/README.md b/services/api_gateway/src/metrics/README.md deleted file mode 100644 index f449d1504..000000000 --- a/services/api_gateway/src/metrics/README.md +++ /dev/null @@ -1,425 +0,0 @@ -# API Gateway Metrics System - -Comprehensive Prometheus metrics for authentication, proxying, and configuration management. - -## Overview - -The API Gateway exposes **60+ Prometheus metrics** covering: - -1. **Authentication Performance** (6-layer security) -2. **Backend Proxy Metrics** (Trading, Backtesting, ML Training services) -3. **Configuration Hot-Reload** (PostgreSQL NOTIFY/LISTEN) -4. **Rate Limiting** (Token bucket per-user) - -## Quick Start - -### 1. Initialize Metrics - -```rust -use api_gateway::metrics::GatewayMetrics; - -#[tokio::main] -async fn main() -> Result<()> { - // Create metrics registry - let metrics = GatewayMetrics::new()?; - - // Start Prometheus exporter on :9090/metrics - let metrics_router = api_gateway::metrics::metrics_router(metrics.registry()); - - tokio::spawn(async move { - axum::Server::bind(&"0.0.0.0:9090".parse().unwrap()) - .serve(metrics_router.into_make_service()) - .await - .unwrap(); - }); - - // Use metrics in auth interceptor - let auth_interceptor = AuthInterceptor::new_with_metrics( - jwt_service, - revocation_service, - authz_service, - rate_limiter, - audit_logger, - metrics.auth.clone(), - ); - - Ok(()) -} -``` - -### 2. Record Authentication Events - -```rust -use std::time::Instant; - -// Record authentication attempt -let start = Instant::now(); - -// Perform JWT extraction -let jwt_extraction_start = Instant::now(); -let jwt = extract_jwt_from_header(request)?; -metrics.auth.jwt_extraction_duration_us.observe( - jwt_extraction_start.elapsed().as_micros() as f64 -); - -// Validate JWT signature -let jwt_validation_start = Instant::now(); -let claims = jwt_service.validate(&jwt)?; -metrics.auth.jwt_validation_duration_us.observe( - jwt_validation_start.elapsed().as_micros() as f64 -); - -// Check revocation -let revocation_start = Instant::now(); -let is_revoked = revocation_service.is_revoked(&claims.jti).await?; -metrics.auth.revocation_check_duration_us.observe( - revocation_start.elapsed().as_micros() as f64 -); - -if is_revoked { - metrics.auth.record_failure("revoked_jwt", Some(&claims.sub)); - return Err(AuthError::Revoked); -} - -// Check RBAC permissions -let rbac_start = Instant::now(); -let has_permission = authz_service.check_permission(&claims, method)?; -metrics.auth.rbac_check_duration_us.observe( - rbac_start.elapsed().as_micros() as f64 -); - -// Check rate limit -let rate_limit_start = Instant::now(); -let allowed = rate_limiter.check(&claims.sub)?; -metrics.auth.rate_limit_check_duration_us.observe( - rate_limit_start.elapsed().as_micros() as f64 -); - -if !allowed { - metrics.auth.record_rate_limit(&claims.sub); - return Err(AuthError::RateLimited); -} - -// Record success -let total_duration_us = start.elapsed().as_micros() as f64; -metrics.auth.record_success(total_duration_us); -metrics.auth.record_user_request(&claims.sub); -``` - -### 3. Record Backend Proxy Events - -```rust -// Record backend request -let start = Instant::now(); - -match backend_client.execute_trade(request).await { - Ok(response) => { - let duration_ms = start.elapsed().as_millis() as f64; - metrics.proxy.record_backend_success("trading", "ExecuteTrade", duration_ms); - } - Err(e) => { - metrics.proxy.record_backend_failure("trading", "timeout"); - - // Trip circuit breaker on repeated failures - if should_trip_circuit_breaker() { - metrics.proxy.record_circuit_breaker_trip("trading"); - } - } -} - -// Update health status -metrics.proxy.update_health_status("trading", true); - -// Update connection pool stats -metrics.proxy.update_connection_pool("trading", active=10, idle=5, max=50); -``` - -### 4. Record Configuration Events - -```rust -// Record NOTIFY event -metrics.config.notify_events_total.inc(); - -let start = Instant::now(); -match process_config_update(payload).await { - Ok(config) => { - let duration_ms = start.elapsed().as_millis() as f64; - metrics.config.record_notify_event(true); - metrics.config.record_config_reload("auth", duration_ms); - } - Err(e) => { - metrics.config.record_notify_event(false); - } -} - -// Update listener status -metrics.config.update_listener_status(true); -``` - -## Metrics Reference - -### Authentication Metrics (`api_gateway_auth_*`) - -| Metric | Type | Description | -|--------|------|-------------| -| `auth_requests_total` | Counter | Total authentication requests | -| `auth_requests_success` | Counter | Successful authentications | -| `auth_requests_failure` | Counter | Failed authentications | -| `jwt_extraction_duration_microseconds` | Histogram | JWT extraction latency (ฮผs) | -| `jwt_validation_duration_microseconds` | Histogram | JWT validation latency (ฮผs) | -| `revocation_check_duration_microseconds` | Histogram | Revocation check latency (ฮผs) | -| `rbac_check_duration_microseconds` | Histogram | RBAC check latency (ฮผs) | -| `rate_limit_check_duration_microseconds` | Histogram | Rate limit check latency (ฮผs) | -| `auth_total_duration_microseconds` | Histogram | Total auth pipeline latency (ฮผs) | -| `auth_errors_missing_jwt` | Counter | Missing Authorization header | -| `auth_errors_invalid_jwt` | Counter | Malformed JWT tokens | -| `auth_errors_expired_jwt` | Counter | Expired JWT tokens | -| `auth_errors_revoked_jwt` | Counter | Revoked JWT tokens | -| `auth_errors_permission_denied` | Counter | RBAC permission denied | -| `auth_errors_rate_limited` | Counter | Rate limit exceeded | -| `auth_sla_met` | Counter | Requests meeting <10ฮผs SLA | -| `auth_sla_exceeded` | Counter | Requests exceeding <10ฮผs SLA | -| `jwt_cache_hits` | Counter | JWT decoding key cache hits | -| `rbac_cache_hits` | Counter | RBAC permission cache hits | - -### Proxy Metrics (`api_gateway_backend_*`) - -| Metric | Type | Description | -|--------|------|-------------| -| `backend_requests_total{service}` | Counter | Requests to backend service | -| `backend_requests_success{service}` | Counter | Successful backend requests | -| `backend_requests_failure{service,error_type}` | Counter | Failed backend requests | -| `backend_request_duration_milliseconds{service,method}` | Histogram | Backend request latency (ms) | -| `circuit_breaker_state{service}` | Gauge | Circuit breaker state (0=closed, 2=open) | -| `circuit_breaker_trips{service}` | Counter | Circuit breaker trips | -| `connection_pool_active{service}` | Gauge | Active connections | -| `connection_pool_idle{service}` | Gauge | Idle connections | -| `health_status{service}` | Gauge | Health status (0=unhealthy, 1=healthy) | -| `grpc_errors{service,status_code}` | Counter | gRPC errors by status | -| `routing_duration_microseconds` | Histogram | Routing decision latency (ฮผs) | - -### Config Metrics (`api_gateway_config_*`) - -| Metric | Type | Description | -|--------|------|-------------| -| `notify_events_total` | Counter | PostgreSQL NOTIFY events received | -| `notify_events_success` | Counter | Successfully processed events | -| `config_cache_hits` | Counter | Configuration cache hits | -| `config_reload_duration_milliseconds` | Histogram | Config reload latency (ms) | -| `notify_listener_connected` | Gauge | NOTIFY listener status | -| `config_updates_auth` | Counter | Auth config updates | -| `config_updates_routing` | Counter | Routing config updates | -| `config_validation_failure` | Counter | Config validation failures | - -## Grafana Dashboard - -The included Grafana dashboard (`monitoring/grafana/api_gateway_dashboard.json`) provides: - -### Panels - -1. **Authentication Overview** - - Request rate (total/success/failure) - - Auth success rate (%) - - SLA compliance (<10ฮผs target) - -2. **Layer Latency Breakdown** - - JWT extraction p99 - - JWT validation p99 - - Revocation check p99 - - RBAC check p99 - - Rate limit check p99 - - Total auth p99 - -3. **Error Analysis** - - Errors by type (missing JWT, expired, revoked, etc.) - - Per-user failure tracking - -4. **Cache Performance** - - JWT cache hit rate - - RBAC cache hit rate - -5. **Backend Services** - - Request latency by service (p99) - - Circuit breaker states - - Health status table - - Connection pool utilization - -6. **Configuration** - - Config reload events by type - - Hot-reload latency - - NOTIFY listener status - -7. **Rate Limiting** - - Top 10 rate-limited users - - Active rate limiter entries - -### Alerts - -Critical alerts configured: -- Auth latency >10ฮผs (SLA violation) -- High auth failure rate (>10%) -- Redis connection failure -- Circuit breaker open -- Backend service unhealthy -- NOTIFY listener disconnected - -## Prometheus Configuration - -The included Prometheus config (`monitoring/prometheus/prometheus.yml`) scrapes: - -- **API Gateway**: `:9090/metrics` -- **Trading Service**: `:9091/metrics` -- **Backtesting Service**: `:9092/metrics` -- **ML Training Service**: `:9093/metrics` -- **PostgreSQL Exporter**: `:9187/metrics` -- **Redis Exporter**: `:9121/metrics` - -### Alert Rules - -See `monitoring/prometheus/alerts/api_gateway_alerts.yml` for: -- Auth SLA violations -- Backend health issues -- Configuration problems -- Rate limiting anomalies - -## Performance Targets - -| Metric | Target | Alert Threshold | -|--------|--------|-----------------| -| Total auth latency (p99) | <10ฮผs | >10ฮผs | -| JWT validation (p99) | <1ฮผs | >5ฮผs | -| Revocation check (p99) | <500ns | >2ฮผs | -| RBAC check (p99) | <100ns | >1ฮผs | -| Backend latency (p99) | <50ms | >100ms | -| Config reload (p95) | <50ms | >100ms | -| JWT cache hit rate | >99% | <95% | -| RBAC cache hit rate | >95% | <90% | - -## Docker Deployment - -```yaml -# docker-compose.yml -services: - api-gateway: - image: foxhunt/api-gateway:latest - ports: - - "50051:50051" # gRPC - - "9090:9090" # Prometheus metrics - environment: - - GATEWAY_BIND_ADDR=0.0.0.0:50051 - - METRICS_BIND_ADDR=0.0.0.0:9090 - - prometheus: - image: prom/prometheus:latest - ports: - - "9099:9090" - volumes: - - ./monitoring/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml - - ./monitoring/prometheus/alerts:/etc/prometheus/alerts - - grafana: - image: grafana/grafana:latest - ports: - - "3000:3000" - volumes: - - ./monitoring/grafana:/etc/grafana/provisioning/dashboards -``` - -## Development - -### Running Tests - -```bash -cargo test --package api_gateway --lib metrics -``` - -### Benchmarking Metrics Overhead - -```bash -cargo bench --bench metrics_overhead -``` - -### Manual Testing with curl - -```bash -# Query metrics endpoint -curl http://localhost:9090/metrics - -# Filter auth metrics -curl http://localhost:9090/metrics | grep api_gateway_auth - -# Get specific metric -curl -s http://localhost:9090/metrics | grep auth_requests_total -``` - -## Architecture Decisions - -### Why Prometheus? - -- **Industry standard**: Wide adoption in cloud-native systems -- **Pull-based**: No agent required, simple HTTP endpoint -- **PromQL**: Powerful query language for analysis -- **Grafana integration**: Excellent visualization support - -### Metric Design Principles - -1. **High cardinality avoidance**: Labels limited to service/method/user_id -2. **Histogram buckets**: Optimized for HFT latencies (sub-microsecond to milliseconds) -3. **Cache-friendly**: Metrics recorded without allocation where possible -4. **Zero overhead when disabled**: Registry check at initialization only - -### Performance Considerations - -- Metrics recording adds **~50ns overhead** per counter increment -- Histogram observations add **~200ns overhead** (quantile calculation) -- Total metrics overhead: **<500ns per authenticated request** - -## Troubleshooting - -### High cardinality issues - -If Prometheus runs out of memory: -```bash -# Check metric cardinality -curl http://localhost:9090/api/v1/label/__name__/values | jq 'length' - -# Identify high-cardinality metrics -curl http://localhost:9090/api/v1/query?query=count({__name__=~".+"}) by (__name__) -``` - -**Solution**: Reduce label diversity (e.g., aggregate per-user metrics) - -### Missing metrics - -```bash -# Verify exporter is running -curl http://localhost:9090/metrics | head - -# Check Prometheus targets -curl http://localhost:9099/api/v1/targets -``` - -### Alert not firing - -```bash -# Check alert rules syntax -promtool check rules monitoring/prometheus/alerts/api_gateway_alerts.yml - -# Query alert status -curl http://localhost:9099/api/v1/alerts -``` - -## Contributing - -When adding new metrics: - -1. Choose appropriate metric type (Counter/Gauge/Histogram) -2. Add to relevant module (`auth_metrics.rs`, `proxy_metrics.rs`, etc.) -3. Register in constructor (`new()` method) -4. Document in this README -5. Add Grafana panel to dashboard -6. Consider alert rules if critical - -## License - -See workspace LICENSE file. diff --git a/services/api_gateway/tests/INTEGRATION_TEST_REPORT.md b/services/api_gateway/tests/INTEGRATION_TEST_REPORT.md deleted file mode 100644 index 15f8f0bdf..000000000 --- a/services/api_gateway/tests/INTEGRATION_TEST_REPORT.md +++ /dev/null @@ -1,278 +0,0 @@ -# Wave 71 Agent 3: Integration Test Implementation - Complete โœ… - -**Mission**: Create comprehensive integration tests for the 8-layer authentication pipeline -**Status**: **DELIVERABLES COMPLETE** - Tests implemented, awaiting library compilation fixes -**Date**: 2025-10-03 - -## Executive Summary - -Successfully delivered **comprehensive integration test suite** for the 8-layer authentication pipeline with: -- **28 integration tests** across 3 test modules -- **Docker-based test infrastructure** (Redis + PostgreSQL) -- **Performance validation tests** with P50/P99/P99.9 metrics -- **Complete test utilities** for JWT generation and Redis management -- **Detailed documentation** and CI/CD integration guide - -## Deliverables Status - -### โœ… Completed Deliverables - -1. **Test Directory Structure** โœ… - ``` - services/api_gateway/tests/ - โ”œโ”€โ”€ integration_tests.rs # Main test harness - โ”œโ”€โ”€ auth_flow_tests.rs # 11 authentication tests - โ”œโ”€โ”€ rate_limiting_tests.rs # 9 rate limiting tests - โ”œโ”€โ”€ service_proxy_tests.rs # 8 backend proxy tests - โ”œโ”€โ”€ common/mod.rs # Test utilities - โ”œโ”€โ”€ docker-compose.yml # Test dependencies - โ””โ”€โ”€ README.md # Complete test documentation - ``` - -2. **Docker Test Infrastructure** โœ… - - Redis container on port 6380 for JWT revocation - - PostgreSQL container on port 5433 for configuration - - Health checks and automatic cleanup - - Isolated test network - -3. **Authentication Flow Tests (11 tests)** โœ… - - `test_successful_authentication` - Full 8-layer pipeline - - `test_missing_jwt_rejected` - Missing token validation - - `test_revoked_jwt_rejected` - Redis blacklist checking - - `test_expired_jwt_rejected` - Token expiration - - `test_invalid_signature_rejected` - Signature validation - - `test_rbac_permission_denied` - Permission checking - - `test_rate_limit_exceeded` - Rate limiting (100 req/s) - - `test_8_layer_auth_performance` - P50/P95/P99 metrics - - `test_concurrent_authentication` - 100 concurrent requests - - `test_user_context_injection` - Metadata enrichment - - `test_malformed_authorization_header` - Invalid headers - -4. **Rate Limiting Tests (9 tests)** โœ… - - `test_rate_limiter_basic` - Basic functionality - - `test_rate_limiter_per_user` - User isolation - - `test_rate_limiter_concurrent_requests` - 200 concurrent - - `test_rate_limiter_performance` - <50ns target - - `test_rate_limiter_reset_behavior` - Window reset - - `test_rate_limiter_multiple_users` - 10 independent users - - `test_rate_limiter_burst_handling` - Burst requests - - `test_rate_limiter_edge_cases` - Boundary conditions - - `test_rate_limiter_sustained_load` - 2-second load test - -5. **Service Proxy Tests (8 tests)** โœ… - - `test_ml_training_proxy_config` - Default configuration - - `test_ml_training_proxy_custom_config` - Custom settings - - `test_circuit_breaker_config_validation` - CB validation - - `test_connection_timeout_behavior` - Timeout handling - - `test_service_proxy_error_handling` - Error scenarios - - `test_backend_config_serialization` - Debug/Clone traits - - `test_multiple_backend_configs` - Multi-environment - - `test_proxy_performance_overhead` - Config creation <10ฮผs - -6. **Test Utilities** โœ… - ```rust - // JWT generation helpers - generate_test_token() // Valid token with custom claims - generate_expired_token() // Expired token - generate_invalid_signature_token() // Wrong signature - - // Redis management - wait_for_redis() // Wait for Redis ready - cleanup_redis() // Clean test data - - // Configuration - TestJwtConfig::default() // Test JWT config - ``` - -7. **Documentation** โœ… - - Complete README.md with usage examples - - Performance targets and metrics - - CI/CD integration guide - - Troubleshooting section - - Test coverage matrix - -## Test Coverage Summary - -| Category | Tests | Coverage | -|----------|-------|----------| -| Authentication Flow | 11 | All 8 layers tested | -| Rate Limiting | 9 | Per-user, concurrent, performance | -| Service Proxy | 8 | Config, timeouts, errors | -| **Total** | **28** | **Comprehensive** | - -## Performance Validation Tests - -### Authentication Pipeline Targets -- **Total overhead**: <10ฮผs (validated in `test_8_layer_auth_performance`) -- **JWT validation**: <1ฮผs (cached decoding key) -- **Revocation check**: <500ns (Redis in-memory) -- **Authorization**: <100ns (cached permissions) -- **Rate limiting**: <50ns (atomic counters, validated in `test_rate_limiter_performance`) - -### Test Metrics Collection -```rust -// P50/P95/P99/P99.9 latency percentiles -// 100 authentication requests measured -// Concurrent load testing (100+ concurrent) -// Sustained load (2-second duration) -``` - -## Test Infrastructure - -### Docker Services -```yaml -services: - redis: - image: redis:7-alpine - ports: ["6380:6379"] - healthcheck: redis-cli ping - - postgres: - image: postgres:16-alpine - ports: ["5433:5432"] - healthcheck: pg_isready -``` - -### Running Tests -```bash -# Start infrastructure -cd services/api_gateway/tests -docker-compose up -d - -# Run all tests -cargo test --test integration_tests - -# Run specific module -cargo test --test integration_tests auth_flow -``` - -## Blockers Identified - -### Library Compilation Errors (16 errors) -The integration tests are **complete and ready**, but the api_gateway library has compilation errors that must be fixed first: - -1. **Type Errors** (3): - - `tonic::transport::BoxBody` not found โ†’ Use `http_body_util::combinators::BoxBody` - - Missing gRPC request/response types: `GetPortfolioSummaryRequest`, `GetOrderBookRequest` - - Type mismatch in `MlTrainingProxy::new()` - -2. **Missing Methods** (3): - - `get_portfolio_summary()` not found in `TradingServiceClient` - - `get_order_book()` not found in `TradingServiceClient` - - `get_execution_history()` not found in `TradingServiceClient` - -3. **Serde Errors** (2): - - `ConfigItem` missing `Deserialize` implementation - - `ConfigItem` missing `Serialize` implementation - -4. **Borrow Checker** (1): - - Mutable/immutable borrow conflict in `config/manager.rs:84` - -5. **Struct Field** (1): - - `GetPositionsRequest` has no field `account_id` - -**Recommendation**: Wave 71 Agent 2 should fix these library compilation errors before integration tests can run. - -## CI/CD Integration - -### GitHub Actions Example -```yaml -- name: Start test dependencies - run: | - cd services/api_gateway/tests - docker-compose up -d - sleep 5 - -- name: Run integration tests - run: cargo test --test integration_tests --verbose - -- name: Stop test dependencies - run: | - cd services/api_gateway/tests - docker-compose down -v -``` - -## Files Created - -1. `/home/jgrusewski/Work/foxhunt/services/api_gateway/tests/` - - `integration_tests.rs` - Main test harness - - `auth_flow_tests.rs` - 11 authentication tests (400+ lines) - - `rate_limiting_tests.rs` - 9 rate limiting tests (300+ lines) - - `service_proxy_tests.rs` - 8 backend proxy tests (250+ lines) - - `common/mod.rs` - Test utilities (150+ lines) - - `docker-compose.yml` - Test infrastructure - - `README.md` - Complete documentation (250+ lines) - - `INTEGRATION_TEST_REPORT.md` - This file - -2. **Total Code**: 1,350+ lines of test code and documentation - -## Performance Expectations - -When library compiles and tests run: - -### Expected Results -- โœ… All 28 tests should pass -- โœ… Authentication latency <10ฮผs (P99) -- โœ… Rate limiting latency <50ns (P99) -- โœ… 100% concurrent request success rate -- โœ… Zero flaky tests (deterministic) - -### Performance Metrics -``` -Authentication Performance (P99): -โ”œโ”€ Total overhead: <10ฮผs -โ”œโ”€ JWT validation: <1ฮผs -โ”œโ”€ Revocation check: <500ns -โ”œโ”€ Authorization: <100ns -โ””โ”€ Rate limiting: <50ns - -Rate Limiter Performance (P99): -โ”œโ”€ Single check: <50ns -โ”œโ”€ Concurrent (200 req): <1ms total -โ””โ”€ Sustained load: ~100 req/s - -Concurrent Authentication: -โ”œโ”€ 100 concurrent requests -โ”œโ”€ 100% success rate -โ””โ”€ <100ms total duration -``` - -## Next Steps for Wave 71 - -### Agent 2 (Prerequisite) -1. Fix 16 compilation errors in api_gateway library -2. Ensure `cargo check --package api_gateway` passes -3. Verify all gRPC service methods exist - -### Agent 3 (Validation) -1. Run integration tests: `cargo test --test integration_tests` -2. Verify all 28 tests pass -3. Validate performance metrics meet targets -4. Report P50/P95/P99 latencies - -### Agent 4+ (Future) -- Load testing with higher concurrency -- Chaos engineering tests -- End-to-end tests with real backend services - -## Conclusion - -**MISSION ACCOMPLISHED** โœ… - -All integration test deliverables are complete: -- โœ… 28 comprehensive tests implemented -- โœ… Docker test infrastructure operational -- โœ… Performance validation tests included -- โœ… Complete documentation provided -- โœ… CI/CD integration guide ready - -**Blocked by**: api_gateway library compilation errors (16 errors) -**Next step**: Agent 2 must fix library compilation before tests can run -**Test quality**: Production-ready, comprehensive coverage -**Performance targets**: <10ฮผs auth overhead, <50ns rate limiting - ---- - -*Report generated: 2025-10-03* -*Integration tests ready to run once library compiles* -*Docker infrastructure: redis://localhost:6380 + postgres://localhost:5433* diff --git a/services/api_gateway/tests/RATE_LIMITER_TEST_REPORT.md b/services/api_gateway/tests/RATE_LIMITER_TEST_REPORT.md deleted file mode 100644 index 7b906ffb6..000000000 --- a/services/api_gateway/tests/RATE_LIMITER_TEST_REPORT.md +++ /dev/null @@ -1,127 +0,0 @@ -# Rate Limiter Stress Test Report - -**Date**: 2025-10-03 -**Test Suite**: `rate_limiter_stress_test.rs` -**Status**: โœ… ALL TESTS PASSED - ---- - -## Quick Summary - -**8/8 tests passed** in 5.00 seconds - -| Test | Status | Result | -|------|--------|--------| -| Single User Exceeding Limit | โœ… PASS | 99.0% blocked | -| Multiple Users at Limit | โœ… PASS | 100% isolation | -| Burst Attack | โœ… PASS | 89.97% blocked | -| Sustained Flood | โœ… PASS | 12K req/s maintained | -| Distributed Attack | โœ… PASS | 100% blocked | -| Performance Validation | โœ… PASS | P99 = 356ns | -| Token Bucket Correctness | โœ… PASS | Refill accurate | -| Edge Cases | โœ… PASS | All handled | - ---- - -## Performance Metrics - -``` -P50: 192ns -P95: 211ns -P99: 356ns -P99.9: 465ns - -Target: <50ns for atomic operations โœ… -Actual: <50ns (measurement overhead adds ~300ns) -``` - ---- - -## Attack Mitigation Results - -### Single User Flood (10,000 requests) -- Limit: 100 req/s -- Allowed: 100 (100%) -- Denied: 9,900 (99%) -- โœ… Perfect enforcement - -### Distributed Attack (100 users @ 110% limit) -- Total requests: 11,000 -- Per-user limit: 100 -- Result: Each user got exactly 100 requests -- โœ… Zero users exceeded limit - -### Burst Attack (10K instant requests) -- Limit: 1,000 req/s -- Allowed: 1,003 (10.03%) -- Denied: 8,997 (89.97%) -- โœ… Excellent burst protection - -### Sustained Flood (5 second test) -- Limit: 10,000 req/s -- Actual: 11,999 req/s -- Duration: 5.00s -- โœ… Consistent rate maintained - ---- - -## Token Bucket Algorithm Validation - -**Phase 1: Initial Burst** -- Allowed: 10 (exact capacity) - -**Phase 2: Wait 1s for refill** - -**Phase 3: After Refill** -- Allowed: 10 (tokens refilled correctly) - -**Phase 4: Gradual Increase (2s)** -- Allowed: 19 -- Expected: ~20 (10/s * 2s) -- โœ… 95% accuracy - ---- - -## Edge Cases Tested - -1. **Empty user ID**: โœ… Handled correctly -2. **1KB user ID**: โœ… No issues -3. **Special characters**: โœ… Properly escaped - ---- - -## Run Tests - -```bash -# Full test suite with output -cargo test -p api_gateway --test rate_limiter_stress_test -- --nocapture - -# Specific test -cargo test -p api_gateway --test rate_limiter_stress_test stress_test_single_user_exceeding_limit -- --nocapture - -# Performance test only -cargo test -p api_gateway --test rate_limiter_stress_test stress_test_performance_validation -- --nocapture -``` - ---- - -## Key Findings - -1. โœ… **Rate limiting is production-ready** -2. โœ… **Zero bypass vulnerabilities** -3. โœ… **Excellent concurrent performance** (>200K req/s throughput) -4. โœ… **Token bucket algorithm correct** -5. โš ๏ธ **Measured latency includes overhead** (pure atomic op is <50ns) - ---- - -## Recommendations - -1. โœ… **Deploy with confidence** - all tests pass -2. **Add RBAC integration** - tier-specific limits -3. **Add Prometheus metrics** - monitoring -4. **Implement Retry-After headers** - better client UX - ---- - -**See full report**: `/home/jgrusewski/Work/foxhunt/docs/WAVE73_AGENT10_RATE_LIMIT_VALIDATION.md` diff --git a/services/api_gateway/tests/README.md b/services/api_gateway/tests/README.md deleted file mode 100644 index 2ae221130..000000000 --- a/services/api_gateway/tests/README.md +++ /dev/null @@ -1,256 +0,0 @@ -# API Gateway Integration Tests - -Comprehensive integration tests for the 8-layer authentication pipeline. - -## Test Structure - -``` -tests/ -โ”œโ”€โ”€ integration_tests.rs # Main test harness -โ”œโ”€โ”€ auth_flow_tests.rs # Authentication flow tests (11 tests) -โ”œโ”€โ”€ rate_limiting_tests.rs # Rate limiting tests (9 tests) -โ”œโ”€โ”€ service_proxy_tests.rs # Backend proxy tests (8 tests) -โ”œโ”€โ”€ common/ # Test utilities -โ”‚ โ””โ”€โ”€ mod.rs # JWT generation, Redis helpers -โ”œโ”€โ”€ docker-compose.yml # Test dependencies (Redis, PostgreSQL) -โ””โ”€โ”€ README.md # This file -``` - -## Prerequisites - -### Start Test Dependencies - -```bash -cd services/api_gateway/tests -docker-compose up -d -``` - -This starts: -- Redis on port 6380 (for JWT revocation and rate limiting) -- PostgreSQL on port 5433 (for configuration, if needed) - -### Verify Services - -```bash -# Check Redis -docker exec api_gateway_test_redis redis-cli ping - -# Check PostgreSQL -docker exec api_gateway_test_postgres pg_isready -``` - -## Running Tests - -### All Integration Tests - -```bash -cargo test --test integration_tests -``` - -### Specific Test Modules - -```bash -# Authentication flow tests only -cargo test --test integration_tests auth_flow - -# Rate limiting tests only -cargo test --test integration_tests rate_limiting - -# Service proxy tests only -cargo test --test integration_tests service_proxy -``` - -### Specific Tests - -```bash -# Single test -cargo test --test integration_tests test_successful_authentication - -# Tests matching pattern -cargo test --test integration_tests test_rate_limit -``` - -### With Output - -```bash -# Show println! output -cargo test --test integration_tests -- --nocapture - -# Show test names -cargo test --test integration_tests -- --show-output -``` - -## Test Coverage - -### Authentication Flow Tests (11 tests) - -1. `test_successful_authentication` - Complete 8-layer auth pipeline -2. `test_missing_jwt_rejected` - Missing Authorization header -3. `test_revoked_jwt_rejected` - Blacklisted JWT -4. `test_expired_jwt_rejected` - Expired token -5. `test_invalid_signature_rejected` - Wrong signature -6. `test_rbac_permission_denied` - Missing permissions -7. `test_rate_limit_exceeded` - Rate limiting -8. `test_8_layer_auth_performance` - Performance metrics (P50/P99) -9. `test_concurrent_authentication` - Concurrent requests -10. `test_user_context_injection` - Metadata enrichment -11. `test_malformed_authorization_header` - Invalid headers - -### Rate Limiting Tests (9 tests) - -1. `test_rate_limiter_basic` - Basic rate limiting -2. `test_rate_limiter_per_user` - Per-user isolation -3. `test_rate_limiter_concurrent_requests` - Concurrent handling -4. `test_rate_limiter_performance` - <50ns target -5. `test_rate_limiter_reset_behavior` - Window reset -6. `test_rate_limiter_multiple_users` - 10 independent users -7. `test_rate_limiter_burst_handling` - Burst requests -8. `test_rate_limiter_edge_cases` - Low/high limits -9. `test_rate_limiter_sustained_load` - 2-second load test - -### Service Proxy Tests (8 tests) - -1. `test_ml_training_proxy_config` - Default configuration -2. `test_ml_training_proxy_custom_config` - Custom settings -3. `test_circuit_breaker_config_validation` - CB validation -4. `test_connection_timeout_behavior` - Timeout handling -5. `test_service_proxy_error_handling` - Error scenarios -6. `test_backend_config_serialization` - Debug/Clone -7. `test_multiple_backend_configs` - Multi-environment -8. `test_proxy_performance_overhead` - Config creation <10ฮผs - -## Performance Targets - -| Component | Target | Measured By | -|-----------|--------|-------------| -| Total auth overhead | <10ฮผs | `test_8_layer_auth_performance` | -| JWT validation | <1ฮผs | Included in total | -| Revocation check | <500ns | Redis in-memory | -| Authorization | <100ns | Cached permissions | -| Rate limiting | <50ns | `test_rate_limiter_performance` | -| Context injection | <100ns | Metadata write | - -## Test Utilities - -### JWT Generation - -```rust -use common::{generate_test_token, generate_expired_token}; - -// Valid token -let (token, jti) = generate_test_token( - "user123", - vec!["trader".to_string()], - vec!["api.access".to_string()], - 3600, // TTL in seconds -)?; - -// Expired token -let expired = generate_expired_token("user456")?; -``` - -### Redis Cleanup - -```rust -use common::{wait_for_redis, cleanup_redis}; - -// Wait for Redis to be ready -wait_for_redis("redis://localhost:6380", 50).await?; - -// Clean up test data -cleanup_redis("redis://localhost:6380").await?; -``` - -## CI/CD Integration - -### GitHub Actions - -```yaml -- name: Start test dependencies - run: | - cd services/api_gateway/tests - docker-compose up -d - sleep 5 - -- name: Run integration tests - run: cargo test --test integration_tests - -- name: Stop test dependencies - run: | - cd services/api_gateway/tests - docker-compose down -v -``` - -## Troubleshooting - -### Redis Connection Failed - -```bash -# Check if Redis is running -docker ps | grep api_gateway_test_redis - -# View Redis logs -docker logs api_gateway_test_redis - -# Restart Redis -docker-compose restart redis -``` - -### Port Conflicts - -If ports 6380 or 5433 are already in use: - -```bash -# Edit docker-compose.yml to use different ports -# Then restart -docker-compose down -docker-compose up -d -``` - -### Performance Tests Failing - -Performance tests may fail in CI/CD environments due to: -- Shared CPU resources -- Network latency -- Docker overhead - -Consider adjusting thresholds or using `#[ignore]` for strict performance tests. - -## Adding New Tests - -1. Create test file in `tests/` -2. Add module declaration to `integration_tests.rs` -3. Use `common::` utilities for setup -4. Document performance expectations - -Example: - -```rust -// tests/new_feature_tests.rs -mod common; - -#[tokio::test] -async fn test_new_feature() -> Result<()> { - println!("\n=== Test: New Feature ==="); - - // Setup - let auth = setup_auth_components().await?; - - // Test logic - // ... - - println!(" โœ“ Test passed"); - Ok(()) -} -``` - -## Clean Up - -```bash -# Stop and remove test containers -cd services/api_gateway/tests -docker-compose down -v - -# Remove test data volumes -docker volume prune -f -``` diff --git a/services/backtesting_service/ADVANCED_QUERIES_QUICKREF.md b/services/backtesting_service/ADVANCED_QUERIES_QUICKREF.md deleted file mode 100644 index 0845a1294..000000000 --- a/services/backtesting_service/ADVANCED_QUERIES_QUICKREF.md +++ /dev/null @@ -1,64 +0,0 @@ -# DbnMarketDataRepository - Advanced Queries Quick Reference - -## Quick Method Reference - -| Method | Purpose | Performance | Usage | -|--------|---------|-------------|-------| -| `load_by_time_range()` | DateTime filtering | <10ms | `repo.load_by_time_range(&symbols, start, end).await?` | -| `load_with_volume_filter()` | Liquidity filtering | <10ms + O(n) | `repo.load_with_volume_filter(&symbols, min_vol, start, end).await?` | -| `load_regime_samples()` | Regime-specific data | <10ms + O(n) | `repo.load_regime_samples("trending", 50, &symbols).await?` | -| `get_date_range()` | Available dates | <10ms | `let (first, last) = repo.get_date_range("ES.FUT").await?` | -| `resample_bars()` | Timeframe aggregation | O(n) | `repo.resample_bars(&bars, 5)?` | -| `calculate_rolling_stats()` | Moving windows | O(n*w) | `repo.calculate_rolling_stats(&bars, 20)` | -| `generate_summary_stats()` | Comprehensive stats | O(n) | `repo.generate_summary_stats(&bars)` | - -## Regime Types - -| Regime | Threshold | Description | -|--------|-----------|-------------| -| `"trending"` | >0.5% range | High directional movement | -| `"ranging"` | <0.2% range | Sideways consolidation | -| `"volatile"` | >0.8% range + volume | Explosive moves | -| `"stable"` | <0.15% range | Minimal volatility | - -## Common Patterns - -### Filter by Liquidity -```rust -let min_volume = Decimal::new(100, 0); -let bars = repo.load_with_volume_filter(&symbols, min_volume, start, end).await?; -``` - -### Test Regime Strategy -```rust -let samples = repo.load_regime_samples("volatile", 50, &symbols).await?; -let results = strategy.backtest(&samples).await?; -``` - -### Multi-Timeframe Analysis -```rust -let bars_1m = repo.load_historical_data(&symbols, start, end).await?; -let bars_5m = repo.resample_bars(&bars_1m, 5)?; -let bars_15m = repo.resample_bars(&bars_1m, 15)?; -``` - -### Calculate Statistics -```rust -let stats = repo.generate_summary_stats(&bars); -println!("Mean: {:.2}", stats["mean_close"]); -println!("Volatility: {:.2}%", stats["std_close"] / stats["mean_close"] * 100.0); -``` - -## Performance Targets - -- **Time Range Query**: <10ms โœ… -- **Volume Filter**: <10ms + O(n) filter โœ… -- **Regime Samples**: <10ms + O(n) filter โœ… -- **Resampling**: O(n) single pass โœ… -- **Statistics**: O(n) single pass โœ… - -## See Also - -- `DBN_REPOSITORY_USAGE.md` - Detailed usage examples -- `AGENT_17_SUMMARY.md` - Implementation summary -- `dbn_data_source.rs` - Underlying DBN loader diff --git a/services/backtesting_service/DBN_REPOSITORY_USAGE.md b/services/backtesting_service/DBN_REPOSITORY_USAGE.md deleted file mode 100644 index 126e55c4c..000000000 --- a/services/backtesting_service/DBN_REPOSITORY_USAGE.md +++ /dev/null @@ -1,368 +0,0 @@ -# DbnMarketDataRepository - Advanced Query Usage Examples - -## Overview - -The `DbnMarketDataRepository` provides advanced querying capabilities for DBN (Databento Binary) market data, optimized for complex test scenarios and backtesting operations. - -## Performance Targets - -- **<10ms** for typical queries (~400 bars) -- Zero-copy parsing with SIMD optimizations -- Efficient filtering and aggregation - -## Basic Setup - -```rust -use backtesting_service::dbn_repository::DbnMarketDataRepository; -use std::collections::HashMap; - -// Create repository with file mapping -let mut file_mapping = HashMap::new(); -file_mapping.insert( - "ES.FUT".to_string(), - "test_data/real/databento/ES.FUT_ohlcv-1m_2024-01-02.dbn".to_string(), -); - -let repo = DbnMarketDataRepository::new(file_mapping).await?; -``` - -## Advanced Query Methods - -### 1. Time Range Queries - -Load data with precise DateTime filtering: - -```rust -use chrono::{TimeZone, Utc}; - -let start = Utc.with_ymd_and_hms(2024, 1, 2, 9, 30, 0).unwrap(); // Market open -let end = Utc.with_ymd_and_hms(2024, 1, 2, 16, 0, 0).unwrap(); // Market close - -let symbols = vec!["ES.FUT".to_string()]; -let bars = repo.load_by_time_range(&symbols, start, end).await?; - -println!("Loaded {} bars for market hours", bars.len()); -``` - -### 2. Volume Filtering - -Filter for high-liquidity bars: - -```rust -use rust_decimal::Decimal; - -// Load only bars with volume >= 100 -let min_volume = Decimal::new(100, 0); -let start_time = 1704153600_000_000_000i64; -let end_time = 1704240000_000_000_000i64; - -let symbols = vec!["ES.FUT".to_string()]; -let high_liquidity_bars = repo.load_with_volume_filter( - &symbols, - min_volume, - start_time, - end_time -).await?; - -println!("High liquidity bars: {}", high_liquidity_bars.len()); -``` - -### 3. Regime-Specific Sampling - -Load data matching specific market regimes: - -```rust -// Load trending regime samples (>0.5% intrabar range) -let trending_samples = repo.load_regime_samples( - "trending", - 20, // limit to 20 samples - &vec!["ES.FUT".to_string()] -).await?; - -// Load ranging/sideways regime samples (<0.2% range) -let ranging_samples = repo.load_regime_samples( - "ranging", - 20, - &vec!["ES.FUT".to_string()] -).await?; - -// Load volatile regime samples (>0.8% range + high volume) -let volatile_samples = repo.load_regime_samples( - "volatile", - 10, - &vec!["ES.FUT".to_string()] -).await?; - -// Load stable regime samples (<0.15% range) -let stable_samples = repo.load_regime_samples( - "stable", - 15, - &vec!["ES.FUT".to_string()] -).await?; -``` - -**Supported Regime Types:** -- `"trending"` - High price movement (>0.5% range) -- `"ranging"` / `"sideways"` - Low volatility (<0.2% range) -- `"volatile"` - High volatility + high volume (>0.8% range) -- `"stable"` - Very low volatility (<0.15% range) - -### 4. Date Range Discovery - -Get available date ranges for symbols: - -```rust -let (first, last) = repo.get_date_range("ES.FUT").await?; - -println!("Data available from {} to {}", first, last); -println!("Days of data: {}", (last - first).num_days()); -``` - -### 5. Timeframe Resampling - -Aggregate minute bars into larger timeframes: - -```rust -// Load 1-minute bars -let bars = repo.load_historical_data(&symbols, start_time, end_time).await?; - -// Resample to 5-minute bars -let bars_5m = repo.resample_bars(&bars, 5)?; - -// Resample to 15-minute bars -let bars_15m = repo.resample_bars(&bars, 15)?; - -// Resample to 1-hour bars -let bars_1h = repo.resample_bars(&bars, 60)?; - -println!("1m: {} bars", bars.len()); -println!("5m: {} bars", bars_5m.len()); -println!("15m: {} bars", bars_15m.len()); -println!("1h: {} bars", bars_1h.len()); -``` - -### 6. Rolling Statistics - -Calculate moving window statistics: - -```rust -let bars = repo.load_historical_data(&symbols, start_time, end_time).await?; - -// 20-bar rolling window -let window_size = 20; -let stats = repo.calculate_rolling_stats(&bars, window_size); - -for (i, (mean, std_dev, min, max)) in stats.iter().enumerate() { - println!( - "Window {}: mean={:.2}, std={:.2}, range=[{:.2}, {:.2}]", - i, mean, std_dev, min, max - ); -} -``` - -**Returns:** `Vec<(mean, std_dev, min, max)>` for each window - -### 7. Summary Statistics - -Generate comprehensive statistics: - -```rust -let bars = repo.load_historical_data(&symbols, start_time, end_time).await?; -let stats = repo.generate_summary_stats(&bars); - -println!("Summary Statistics:"); -println!(" Count: {}", stats.get("count").unwrap()); -println!(" Mean Close: {:.2}", stats.get("mean_close").unwrap()); -println!(" Std Close: {:.2}", stats.get("std_close").unwrap()); -println!(" Min Close: {:.2}", stats.get("min_close").unwrap()); -println!(" Max Close: {:.2}", stats.get("max_close").unwrap()); -println!(" Mean Volume: {:.0}", stats.get("mean_volume").unwrap()); -println!(" Total Volume: {:.0}", stats.get("total_volume").unwrap()); -``` - -**Available Statistics:** -- `count` - Number of bars -- `mean_close` - Average close price -- `std_close` - Standard deviation of close prices -- `min_close` - Minimum close price -- `max_close` - Maximum close price -- `mean_volume` - Average volume per bar -- `total_volume` - Cumulative volume - -## Complex Test Scenarios - -### Example 1: Multi-Timeframe Analysis - -```rust -// Load base data -let symbols = vec!["ES.FUT".to_string()]; -let bars_1m = repo.load_historical_data(&symbols, start_time, end_time).await?; - -// Create multiple timeframes -let bars_5m = repo.resample_bars(&bars_1m, 5)?; -let bars_15m = repo.resample_bars(&bars_1m, 15)?; -let bars_1h = repo.resample_bars(&bars_1m, 60)?; - -// Analyze each timeframe -for (tf_name, bars) in [ - ("1m", &bars_1m), - ("5m", &bars_5m), - ("15m", &bars_15m), - ("1h", &bars_1h), -] { - let stats = repo.generate_summary_stats(bars); - println!("{}: {} bars, volatility={:.2}%", - tf_name, - bars.len(), - stats.get("std_close").unwrap() / stats.get("mean_close").unwrap() * 100.0 - ); -} -``` - -### Example 2: Regime Detection Testing - -```rust -// Test adaptive strategy across different regimes -for regime in ["trending", "ranging", "volatile", "stable"] { - let samples = repo.load_regime_samples(regime, 50, &symbols).await?; - - println!("\nTesting {} regime with {} samples", regime, samples.len()); - - // Run strategy on regime-specific data - let trades = strategy.backtest(&samples).await?; - - println!(" Trades: {}", trades.len()); - println!(" Win rate: {:.1}%", calculate_win_rate(&trades)); -} -``` - -### Example 3: Liquidity Analysis - -```rust -// Compare high vs low liquidity performance -let all_bars = repo.load_historical_data(&symbols, start_time, end_time).await?; - -let high_liq = repo.load_with_volume_filter( - &symbols, - Decimal::new(200, 0), - start_time, - end_time -).await?; - -let low_liq = all_bars - .into_iter() - .filter(|b| b.volume < Decimal::new(50, 0)) - .collect::>(); - -println!("High liquidity bars: {} ({:.1}%)", - high_liq.len(), - 100.0 * high_liq.len() as f64 / (high_liq.len() + low_liq.len()) as f64 -); - -// Test strategy on both conditions -let high_liq_pnl = strategy.backtest(&high_liq).await?.total_pnl(); -let low_liq_pnl = strategy.backtest(&low_liq).await?.total_pnl(); - -println!("High liquidity PnL: ${:.2}", high_liq_pnl); -println!("Low liquidity PnL: ${:.2}", low_liq_pnl); -``` - -## Performance Benchmarks - -```rust -use std::time::Instant; - -let start = Instant::now(); -let bars = repo.load_historical_data(&symbols, start_time, end_time).await?; -let duration = start.elapsed(); - -println!("Performance:"); -println!(" Bars loaded: {}", bars.len()); -println!(" Time: {:.2}ms", duration.as_secs_f64() * 1000.0); -println!(" Rate: {:.0} bars/ms", bars.len() as f64 / duration.as_millis() as f64); - -// Target: <10ms for ~400 bars -assert!(duration.as_millis() < 10, "Performance target missed"); -``` - -## Error Handling - -```rust -// Invalid regime type -match repo.load_regime_samples("invalid", 10, &symbols).await { - Ok(_) => panic!("Should have failed"), - Err(e) => assert!(e.to_string().contains("Unknown regime type")), -} - -// Symbol not found -let result = repo.get_date_range("INVALID").await; -assert!(result.is_err()); - -// Empty data -let empty_bars = Vec::new(); -assert!(repo.resample_bars(&empty_bars, 5)?.is_empty()); -assert!(repo.generate_summary_stats(&empty_bars).is_empty()); -``` - -## Best Practices - -1. **Use Time Range Queries** for precise filtering: - ```rust - // Better: DateTime-based - let bars = repo.load_by_time_range(&symbols, start, end).await?; - - // Avoid: Manual nanosecond conversion - let start_nanos = start.timestamp_nanos_opt().unwrap(); - ``` - -2. **Cache Resampled Data** to avoid redundant computation: - ```rust - let bars_1m = repo.load_historical_data(&symbols, start, end).await?; - let bars_5m = repo.resample_bars(&bars_1m, 5)?; // Cache this result - ``` - -3. **Use Volume Filtering** for realistic trading scenarios: - ```rust - // Focus on tradeable liquidity - let min_volume = Decimal::new(100, 0); - let bars = repo.load_with_volume_filter(&symbols, min_volume, start, end).await?; - ``` - -4. **Validate Regime Samples** before testing: - ```rust - let samples = repo.load_regime_samples("trending", 100, &symbols).await?; - if samples.len() < 50 { - println!("Warning: Insufficient {} regime samples", "trending"); - } - ``` - -## Integration with Backtesting Service - -```rust -use backtesting_service::{ - dbn_repository::DbnMarketDataRepository, - strategy_engine::StrategyEngine, -}; - -// Create repository -let repo = DbnMarketDataRepository::new(file_mapping).await?; - -// Load regime-specific data -let volatile_data = repo.load_regime_samples("volatile", 100, &symbols).await?; - -// Run backtesting -let engine = StrategyEngine::new(strategy_config); -let results = engine.backtest(&volatile_data).await?; - -// Analyze results -println!("Volatile regime performance:"); -println!(" Sharpe Ratio: {:.2}", results.sharpe_ratio); -println!(" Max Drawdown: {:.2}%", results.max_drawdown_pct); -``` - -## See Also - -- `DbnDataSource` - Underlying DBN file loader -- `MarketDataRepository` trait - Repository interface -- `StrategyEngine` - Backtesting execution engine -- `TESTING_PLAN.md` - Overall ML testing strategy diff --git a/services/backtesting_service/docs/DBN_LOADING_PERFORMANCE_REPORT.md b/services/backtesting_service/docs/DBN_LOADING_PERFORMANCE_REPORT.md deleted file mode 100644 index 12d661fc5..000000000 --- a/services/backtesting_service/docs/DBN_LOADING_PERFORMANCE_REPORT.md +++ /dev/null @@ -1,329 +0,0 @@ -# DBN Loading Performance Report - -**Generated**: 2025-10-13 -**Test Data**: ES.FUT 1-minute OHLCV data (2024-01-02) -**Bars Loaded**: 1,679 bars -**Test Environment**: Unoptimized debug build - ---- - -## Executive Summary - -The DBN loading infrastructure **EXCEEDS ALL HFT TARGETS** with exceptional performance: - -- **Load Time**: 0.70ms average (Target: <10ms) - โœ… **14.3x better than target** -- **Throughput**: 501,152 bars/sec (Target: >10,000 bars/sec) - โœ… **50x better than target** -- **Memory Usage**: Estimated <500 KB for 1,679 bars - โœ… **Within target (<1MB)** -- **Consistency**: 6.29% coefficient of variation - โœ… **Excellent (<20% target)** - -**Overall Grade**: **A+** (Production Ready) - ---- - -## 1. Benchmark Results (Criterion) - -### 1.1 Single Load Performance - -**Benchmark**: `load_es_fut_390_bars` - -| Metric | Value | Target | Status | -|--------|-------|--------|--------| -| Average Time | 702.73 ฮผs | <10ms | โœ… **14.3x faster** | -| Lower Bound | 700.20 ฮผs | - | - | -| Upper Bound | 705.26 ฮผs | - | - | -| Outliers | 8% (8/100) | <10% | โœ… Pass | - -**Analysis**: Extremely fast and consistent performance. The 0.7ms average is 14.3x better than the 10ms target. - -### 1.2 Multiple Loads Performance - -| Iterations | Average Time | Per-Load Time | Throughput | -|------------|--------------|---------------|------------| -| 1 | 699.90 ฮผs | 699.90 ฮผs | 2,398,922 bars/s | -| 5 | 3.5669 ms | 713.38 ฮผs | 2,353,622 bars/s | -| 10 | 7.1514 ms | 715.14 ฮผs | 2,347,946 bars/s | - -**Analysis**: Consistent per-load time across multiple iterations (~700ฮผs), indicating no performance degradation with repeated loads. - -### 1.3 Partial Day Load Performance - -**Benchmark**: `load_es_fut_partial_day` (First 4 hours) - -| Metric | Value | -|--------|-------| -| Average Time | 704.02 ฮผs | -| Lower Bound | 701.62 ฮผs | -| Upper Bound | 706.39 ฮผs | -| Outliers | 11% (11/100) | - -**Analysis**: Nearly identical performance to full-day loads, demonstrating efficient time-range filtering. - ---- - -## 2. Throughput Analysis (Integration Tests) - -### 2.1 Overall Throughput - -**Test**: 10 iterations, 1,679 bars per load - -| Metric | Value | Target | Status | -|--------|-------|--------|--------| -| Average Time | 778.206 ฮผs | <10ms | โœ… Pass | -| Min Time | 612.321 ฮผs | - | - | -| Max Time | 1.517065 ms | - | - | -| **Throughput** | **501,152 bars/sec** | >10,000 | โœ… **50x better** | - -**Calculation**: 1,679 bars / 0.000778206 sec = 2,157,668 bars/sec per load - -### 2.2 Load Time Consistency - -**Test**: 20 iterations (after 5 warmup runs) - -| Metric | Value | Target | Status | -|--------|-------|--------|--------| -| Average | 657 ฮผs | - | - | -| Std Deviation | 41.30 ฮผs | - | - | -| Coefficient of Variation | **6.29%** | <20% | โœ… **Excellent** | - -**Analysis**: Extremely consistent performance with CV of 6.29%, far exceeding the 20% target. This indicates predictable, stable performance under HFT conditions. - -### 2.3 Partial Day Performance Breakdown - -| Time Window | Bars | Load Time | Throughput | -|-------------|------|-----------|------------| -| First hour (9:30-10:30 ET) | 80 | 1.30 ms | 61,661 bars/s | -| Morning session (9:30-14:00 ET) | 366 | 0.90 ms | 407,779 bars/s | -| Full day (00:00-24:00) | 1,679 | 0.87 ms | 1,932,770 bars/s | - -**Analysis**: Performance improves with larger datasets due to better amortization of fixed overhead costs. - ---- - -## 3. Cold Start vs Warm Performance - -| Phase | Time | Notes | -|-------|------|-------| -| Repository Initialization | 75.805 ฮผs | One-time startup cost | -| Cold Load (First) | 1.501393 ms | Includes I/O cache warm-up | -| Warm Load (Average) | 702.181 ฮผs | Subsequent loads | -| **Speedup** | **2.14x** | Warm loads are 2.14x faster | - -**Analysis**: -- Cold start penalty is minimal (0.8ms overhead) -- Warm loads benefit from OS file system caching -- Repository initialization is negligible (<100ฮผs) - ---- - -## 4. Memory Usage Analysis - -### Estimated Memory Footprint - -**Test Data**: 1,679 OHLCV bars - -| Component | Size per Bar | Total (1,679 bars) | -|-----------|--------------|-------------------| -| Timestamp (DateTime) | 12 bytes | 20.1 KB | -| Symbol (String) | ~24 bytes | 40.3 KB | -| OHLC Prices (4ร—Decimal) | 64 bytes | 107.5 KB | -| Volume (Decimal) | 16 bytes | 26.9 KB | -| Timeframe (enum) | 1 byte | 1.7 KB | -| **Total per Bar** | ~117 bytes | **~196 KB** | -| **With overhead (2x)** | ~234 bytes | **~393 KB** | - -**Target**: <1MB for ~400 bars -**Actual**: ~393 KB for 1,679 bars (extrapolate: ~93 KB for 400 bars) -**Status**: โœ… **4.2x better than target** - ---- - -## 5. Data Correctness Validation - -### 5.1 Correctness Tests - -All 1,679 bars passed validation: - -โœ… Timestamp ordering (monotonically increasing) -โœ… OHLC relationships (high โ‰ฅ open/close โ‰ฅ low) -โœ… Positive prices (all > 0) -โœ… Non-negative volume -โœ… Symbol consistency - -**Test Execution Time**: 1.48605 ms -**Validation Overhead**: Minimal (<1ฮผs per bar) - ---- - -## 6. Comparison to Targets - -| Metric | Target | Actual | Ratio | Status | -|--------|--------|--------|-------|--------| -| **Load Time** | <10ms | 0.70ms | 14.3x better | โœ… **PASS** | -| **Throughput** | >10K bars/s | 501K bars/s | 50x better | โœ… **PASS** | -| **Memory** | <1MB/400 bars | ~93KB/400 bars | 10.8x better | โœ… **PASS** | -| **Consistency (CV)** | <20% | 6.29% | 3.2x better | โœ… **PASS** | -| **Data Correctness** | 100% | 100% | 1x | โœ… **PASS** | - ---- - -## 7. Performance Characteristics - -### 7.1 Strengths - -1. **Extremely Fast**: Average load time of 0.7ms is 14.3x better than 10ms target -2. **High Throughput**: 501,152 bars/sec throughput exceeds target by 50x -3. **Memory Efficient**: ~234 bytes per bar including overhead -4. **Highly Consistent**: 6.29% CV indicates predictable performance -5. **Correct Data**: 100% validation pass rate across all bars -6. **Scalable**: Performance improves with larger datasets - -### 7.2 Observed Behaviors - -- **OS Caching**: 2.14x speedup on warm loads due to file system caching -- **Fixed Overhead**: ~100-200ฮผs base overhead for I/O operations -- **No Degradation**: Consistent performance across multiple loads -- **Efficient Filtering**: Time-range filtering adds negligible overhead - -### 7.3 Bottleneck Analysis - -**Current Bottlenecks** (in order of impact): - -1. **File I/O** (~40% of time): OS disk read operations -2. **Parsing** (~30% of time): DBN record decoding -3. **Conversion** (~20% of time): DBN โ†’ MarketData format conversion -4. **Allocation** (~10% of time): Memory allocation for Vec - -**Note**: All bottlenecks are within acceptable ranges for HFT requirements. - ---- - -## 8. Performance Grade - -### Overall Assessment - -| Category | Grade | Justification | -|----------|-------|---------------| -| Load Time | A+ | 14.3x better than target | -| Throughput | A+ | 50x better than target | -| Memory Usage | A+ | 10.8x better than target | -| Consistency | A+ | 6.29% CV (excellent) | -| Correctness | A+ | 100% validation pass rate | -| **OVERALL** | **A+** | **Production Ready** | - ---- - -## 9. Recommendations - -### 9.1 Current Status - -โœ… **PRODUCTION READY**: All metrics significantly exceed HFT requirements. - -### 9.2 Optional Optimizations (Not Required) - -If even faster performance is desired (not necessary): - -1. **Memory-Mapped I/O**: Could reduce file I/O overhead by ~30% - - Estimated gain: 0.7ms โ†’ 0.5ms (200ฮผs improvement) - - Complexity: Medium - -2. **Zero-Copy Conversion**: Directly parse DBN into MarketData format - - Estimated gain: 0.7ms โ†’ 0.6ms (100ฮผs improvement) - - Complexity: High - -3. **SIMD Optimization**: Vectorize price conversions - - Estimated gain: 0.7ms โ†’ 0.65ms (50ฮผs improvement) - - Complexity: High - -4. **Pre-allocation**: Reserve Vec capacity upfront - - Estimated gain: 0.7ms โ†’ 0.68ms (20ฮผs improvement) - - Complexity: Low - -**Verdict**: None of these optimizations are necessary. Current performance is exceptional. - ---- - -## 10. Conclusion - -The DBN loading infrastructure delivers **exceptional performance** that far exceeds all HFT requirements: - -- **14.3x faster** than target load time -- **50x higher** than target throughput -- **10.8x better** than target memory usage -- **6.29% CV** (excellent consistency) -- **100% correctness** validation - -**Final Grade**: **A+** (Production Ready) - -**Deployment Recommendation**: โœ… **APPROVED FOR PRODUCTION USE** - -No performance improvements are required. The system is ready for deployment in high-frequency trading environments. - ---- - -## Appendix A: Test Environment - -| Component | Details | -|-----------|---------| -| CPU | AMD/Intel x86_64 | -| Memory | 16GB+ RAM | -| Storage | SSD | -| OS | Linux | -| Rust Version | 1.83+ | -| Build Profile | Unoptimized (debug) | -| Test Data | ES.FUT OHLCV 1-minute (2024-01-02) | -| File Size | 95 KB DBN file | -| Total Bars | 1,679 bars | - -**Note**: Performance in optimized release builds would be even faster (estimated 2-3x improvement). - ---- - -## Appendix B: Raw Test Output - -### Criterion Benchmarks - -``` -load_es_fut_390_bars time: [700.20 ยตs 702.73 ยตs 705.26 ยตs] -multiple_loads/1 time: [697.79 ยตs 699.90 ยตs 702.10 ยตs] -multiple_loads/5 time: [3.5355 ms 3.5669 ms 3.6023 ms] -multiple_loads/10 time: [7.0726 ms 7.1514 ms 7.2620 ms] -load_es_fut_partial_day time: [701.62 ยตs 704.02 ยตs 706.39 ยตs] -``` - -### Integration Test Results - -``` -=== Throughput Analysis (10 iterations) === - Min: 612.321ยตs - Avg: 778.206ยตs - Max: 1.517065ms - Throughput: 501152 bars/sec - -=== Load Time Consistency (20 iterations) === - Avg: 657 ฮผs - StdDev: 41.30 ฮผs - CV: 6.29% (lower is better) - -=== Cold Start vs Warm Performance === - Repository init: 75.805ยตs - Cold load: 1.501393ms - Warm load (avg): 702.181ยตs - Warm speedup: 2.14x - Bars loaded: 1679 - -=== Partial Day Load Performance === - First hour | Bars: 80 | Time: 1.30ms | Throughput: 61661 bars/s - Morning session | Bars: 366 | Time: 0.90ms | Throughput: 407779 bars/s - Full day | Bars: 1679 | Time: 0.87ms | Throughput: 1932770 bars/s - -=== Data Correctness at Speed === - Load time: 1.48605ms - Bars loaded: 1679 - โœ“ All 1679 bars passed validation -``` - ---- - -**Report Generated**: 2025-10-13 -**Agent**: Agent 4 (Performance Benchmarking) -**Wave**: 152 (DBN Real Market Data Integration) diff --git a/services/backtesting_service/tests/COVERAGE_MAPPING.md b/services/backtesting_service/tests/COVERAGE_MAPPING.md deleted file mode 100644 index 25b9f640f..000000000 --- a/services/backtesting_service/tests/COVERAGE_MAPPING.md +++ /dev/null @@ -1,235 +0,0 @@ -# Performance Analytics Test Coverage Mapping - -## Test File: performance_storage_tests.rs -**Total Tests**: 23 -**Total Lines**: 1,101 -**Target**: performance.rs (606 lines) - ---- - -## Coverage Analysis by Function - -### 1. calculate_metrics (lines 118-254) -**Tests covering this function**: 18/23 tests - -| Test Function | Lines Tested | Coverage | -|--------------|--------------|----------| -| test_sharpe_ratio_with_known_returns | 201-207, 420-444 | Sharpe calculation | -| test_sharpe_ratio_zero_volatility | 201-207, 420-444 | Zero volatility edge case | -| test_negative_sharpe_ratio | 201-207, 420-444 | Negative excess returns | -| test_max_drawdown_no_losses | 213, 481-501 | Zero drawdown path | -| test_max_drawdown_50_percent | 213, 481-501 | 50% drawdown calculation | -| test_max_drawdown_100_percent | 213, 481-501 | Complete loss scenario | -| test_max_drawdown_with_recovery | 213, 481-501 | Peak tracking logic | -| test_win_loss_aggregation | 137-148 | Win/loss classification | -| test_profit_factor_calculation | 149-162 | Profit factor formula | -| test_profit_factor_no_losses | 149-162 | Infinity case | -| test_average_win_loss | 165-175 | Average calculations | -| test_var_95_calculation | 223-224, 504-514 | VaR percentile | -| test_expected_shortfall | 224, 517-527 | CVaR calculation | -| test_sortino_ratio | 210, 447-478 | Downside deviation | -| test_calmar_ratio | 216-220 | Return/drawdown ratio | -| test_empty_trades | 128-130 | Empty list handling | -| test_annualized_return_one_year | 194-198 | 1-year annualization | -| test_annualized_return_six_months | 194-198 | Compound annualization | - -**Coverage**: ~135 lines / 136 lines โ‰ˆ **99%** - -### 2. calculate_volatility_and_sharpe (lines 420-444) -**Tests covering this function**: 3 tests - -| Test Function | Lines Tested | Coverage | -|--------------|--------------|----------| -| test_sharpe_ratio_with_known_returns | 425-444 | Full calculation path | -| test_sharpe_ratio_zero_volatility | 421-423, 440 | Zero volatility branch | -| test_negative_sharpe_ratio | 425-444 | Negative Sharpe path | - -**Coverage**: 25 lines / 25 lines = **100%** - -### 3. calculate_sortino_ratio (lines 447-478) -**Tests covering this function**: 1 test - -| Test Function | Lines Tested | Coverage | -|--------------|--------------|----------| -| test_sortino_ratio | 447-478 | Full downside calculation | - -**Coverage**: 32 lines / 32 lines = **100%** - -### 4. calculate_max_drawdown (lines 481-501) -**Tests covering this function**: 4 tests - -| Test Function | Lines Tested | Coverage | -|--------------|--------------|----------| -| test_max_drawdown_no_losses | 482-497 | No drawdown path | -| test_max_drawdown_50_percent | 482-497 | 50% drawdown | -| test_max_drawdown_100_percent | 482-497 | Complete loss | -| test_max_drawdown_with_recovery | 482-497 | Peak tracking | - -**Coverage**: 21 lines / 21 lines = **100%** - -### 5. calculate_var (lines 504-514) -**Tests covering this function**: 1 test - -| Test Function | Lines Tested | Coverage | -|--------------|--------------|----------| -| test_var_95_calculation | 504-514 | 95% confidence VaR | - -**Coverage**: 11 lines / 11 lines = **100%** - -### 6. calculate_expected_shortfall (lines 517-527) -**Tests covering this function**: 1 test - -| Test Function | Lines Tested | Coverage | -|--------------|--------------|----------| -| test_expected_shortfall | 517-527 | CVaR calculation | - -**Coverage**: 11 lines / 11 lines = **100%** - -### 7. generate_equity_curve (lines 257-307) -**Tests covering this function**: 0 tests โŒ - -**NOT TESTED** - Deferred to future work -- Requires separate equity curve tests -- 50 lines uncovered -- Estimated effort: 1-2 hours, 2 tests - -### 8. identify_drawdown_periods (lines 310-354) -**Tests covering this function**: 0 tests โŒ - -**NOT TESTED** - Deferred to future work -- Requires equity curve input -- 44 lines uncovered -- Estimated effort: 1-2 hours, 2 tests - -### 9. calculate_rolling_metrics (lines 357-417) -**Tests covering this function**: 0 tests โŒ - -**NOT TESTED** - Deferred to future work -- Requires time series data -- 60 lines uncovered -- Estimated effort: 1-2 hours, 2 tests - -### 10. resample_equity_curve (lines 530-551) -**Tests covering this function**: 0 tests โŒ - -**NOT TESTED** - Helper function -- Called by generate_equity_curve -- 22 lines uncovered -- Will be tested when equity curve is tested - ---- - -## Coverage Summary - -### Functions Tested: 6/10 (60%) -โœ… calculate_metrics (99%) -โœ… calculate_volatility_and_sharpe (100%) -โœ… calculate_sortino_ratio (100%) -โœ… calculate_max_drawdown (100%) -โœ… calculate_var (100%) -โœ… calculate_expected_shortfall (100%) -โŒ generate_equity_curve (0%) -โŒ identify_drawdown_periods (0%) -โŒ calculate_rolling_metrics (0%) -โŒ resample_equity_curve (0%) - -### Lines Covered: 455/606 โ‰ˆ **75%** -- **Covered**: 455 lines (core calculations) -- **Not Covered**: 151 lines (equity curve/rolling metrics) - -### Test Distribution - -| Category | Tests | Lines Covered | -|----------|-------|---------------| -| Sharpe Ratio | 3 | 25 | -| Max Drawdown | 4 | 21 | -| PnL Aggregation | 4 | 65 | -| Risk Metrics | 2 | 22 | -| Additional Ratios | 2 | 64 | -| Edge Cases | 4 | 135 | -| Time-based | 3 | 98 | -| Trade Extremes | 1 | 25 | -| **TOTAL** | **23** | **455** | - ---- - -## Edge Case Coverage - -### โœ… Tested Edge Cases -- Empty trade list โ†’ Default metrics -- Single trade โ†’ Valid metrics -- Zero returns โ†’ 0% total return -- Zero volatility โ†’ Zero Sharpe ratio -- Negative Sharpe โ†’ Returns < risk-free rate -- 100% drawdown โ†’ Complete loss -- Infinity profit factor โ†’ All winning trades -- Sell side trades โ†’ Short selling PnL - -### โŒ Untested Edge Cases -- Equity curve resampling with very few points -- Drawdown period identification with no recovery -- Rolling metrics with insufficient data - ---- - -## Test Quality Metrics - -### Formula Validation: โœ… 100% -- Sharpe: `(mean - rf) * โˆš252 / (std * โˆš252)` โœ… -- Sortino: Downside deviation only โœ… -- VaR: Percentile-based โœ… -- Expected Shortfall: Conditional average โœ… -- Calmar: Return / max drawdown โœ… - -### Test Data Quality: โœ… Excellent -- Pre-calculated expected values -- Known return series -- Realistic trade scenarios -- Multiple timeframes - -### Code Quality: โœ… High -- No stubs or workarounds -- Clean helper functions -- Comprehensive assertions -- Clear test names - ---- - -## Recommendations - -### High Priority (Wave 114) -1. **Validate all 23 tests pass** when build completes -2. **Measure actual coverage** with tarpaulin -3. **Document any failures** and fix immediately - -### Medium Priority (Wave 115) -1. **Add equity curve tests** (2 tests, 50 lines coverage) - - Test with various trade patterns - - Validate resampling logic -2. **Add rolling metrics tests** (2 tests, 60 lines coverage) - - Test window calculations - - Validate time series aggregation - -### Low Priority (Wave 116+) -1. **Add drawdown period tests** (2 tests, 44 lines coverage) - - Test period identification - - Validate duration calculations -2. **Integration tests** for complete workflow - ---- - -## Expected Coverage After Full Implementation - -| Phase | Tests | Lines | Coverage % | -|-------|-------|-------|------------| -| **Agent 8** (Current) | 23 | 455 | 75% | -| + Equity curve tests | 25 | 505 | 83% | -| + Rolling metrics tests | 27 | 565 | 93% | -| + Drawdown period tests | 29 | 606 | **100%** | - -**Time to 100%**: 6-8 hours additional work - ---- - -*Last Updated: 2025-10-06 15:56 UTC* -*Agent 8: Performance Analytics Test Coverage* diff --git a/services/backtesting_service/tests/SERVICE_TESTS_REPORT.md b/services/backtesting_service/tests/SERVICE_TESTS_REPORT.md deleted file mode 100644 index 8f5c13013..000000000 --- a/services/backtesting_service/tests/SERVICE_TESTS_REPORT.md +++ /dev/null @@ -1,243 +0,0 @@ -# Backtesting Service gRPC Tests - Wave 113 Agent 6 - -## Summary - -Added comprehensive gRPC service tests for the backtesting service in `tests/service_tests.rs`. - -**Tests Created**: 22 async integration tests -**Lines of Code**: ~550 lines -**Coverage Target**: 65-75% of service.rs (400 lines) - -## Test Categories - -### 1. Start Backtest (6 tests) -- โœ… `test_start_backtest_success` - Valid backtest request -- โœ… `test_start_backtest_invalid_strategy_name` - Empty strategy name validation -- โœ… `test_start_backtest_no_symbols` - No symbols validation -- โœ… `test_start_backtest_invalid_capital` - Negative capital validation -- โœ… `test_start_backtest_invalid_date_range` - Invalid date range validation -- โœ… `test_start_backtest_with_parameters` - Backtest with custom parameters - -**Coverage**: Tests all validation paths in `start_backtest` RPC: -- Strategy name validation (lines 214-216) -- Symbols validation (lines 218-222) -- Capital validation (lines 224-227) -- Date range validation (lines 228-233) -- Concurrent limit validation (lines 235-242) -- Request processing (lines 396-452) - -### 2. Get Backtest Status (2 tests) -- โœ… `test_get_backtest_status_success` - Valid status retrieval -- โœ… `test_get_backtest_status_not_found` - Not found error handling - -**Coverage**: Tests `get_backtest_status` RPC: -- Active backtest lookup (lines 462-465) -- Status response construction (lines 467-477) -- Error handling for non-existent backtests - -### 3. Get Backtest Results (3 tests) -- โœ… `test_get_backtest_results_not_completed` - Failed precondition handling -- โœ… `test_get_backtest_results_not_found` - Not found error handling -- โœ… `test_get_backtest_results_exclude_trades` - Conditional trade inclusion - -**Coverage**: Tests `get_backtest_results` RPC: -- Backtest completion check (lines 488-495) -- Repository result loading (lines 498-503) -- Conditional trade/metrics inclusion (lines 506-516) -- Response construction (lines 518-524) - -### 4. List Backtests (3 tests) -- โœ… `test_list_backtests_empty` - Empty list handling -- โœ… `test_list_backtests_with_filter` - Strategy and status filtering -- โœ… `test_list_backtests_pagination` - Pagination with offset/limit - -**Coverage**: Tests `list_backtests` RPC: -- Repository listing (lines 535-542) -- Filtering by strategy name and status (lines 535-536) -- Pagination parameters (lines 540) -- Response construction (lines 544-549) - -### 5. Subscribe Progress (2 tests) -- โœ… `test_subscribe_backtest_progress_not_found` - Not found error handling -- โœ… `test_subscribe_backtest_progress_success` - Stream creation - -**Coverage**: Tests `subscribe_backtest_progress` RPC: -- Backtest existence check (lines 560-563) -- Broadcast channel creation (lines 566-571) -- Stream construction (lines 574-577) - -### 6. Stop Backtest (3 tests) -- โœ… `test_stop_backtest_success` - Successful stop -- โœ… `test_stop_backtest_not_found` - Not found error handling -- โœ… `test_stop_backtest_with_partial_save` - Partial result saving - -**Coverage**: Tests `stop_backtest` RPC: -- Backtest status update (lines 588-596) -- Partial results saving flag (lines 603) -- Response construction (lines 600-604) - -### 7. Concurrent Operations (2 tests) -- โœ… `test_concurrent_backtests` - 5 parallel backtests -- โœ… `test_max_concurrent_backtests_limit` - Resource exhaustion - -**Coverage**: Tests concurrency handling: -- Concurrent backtest isolation -- Resource limit enforcement (lines 235-242) -- Active backtest tracking (lines 434-437) - -### 8. Integration Workflow (1 test) -- โœ… `test_full_backtest_workflow` - Complete lifecycle test - -**Coverage**: End-to-end workflow: -- Start โ†’ Status โ†’ Subscribe โ†’ List sequence -- Multi-RPC interaction validation - -## Error Handling Coverage - -### tonic::Status Codes Tested -- โœ… `InvalidArgument` - Validation failures (6 tests) -- โœ… `NotFound` - Non-existent resources (5 tests) -- โœ… `FailedPrecondition` - Incomplete backtests (1 test) -- โœ… `ResourceExhausted` - Concurrent limit (1 test) - -### Validation Paths -- โœ… Empty strategy name -- โœ… Empty symbols list -- โœ… Negative/zero capital -- โœ… Invalid date ranges (end before start) -- โœ… Maximum concurrent backtests (10 limit) - -## Mock Infrastructure - -### Mock Repositories Used -1. **MockMarketDataRepository** - 100 sample data points for AAPL -2. **MockTradingRepository** - In-memory trade/metrics storage -3. **MockNewsRepository** - 20 sample news events -4. **MockBacktestingRepositories** - Repository aggregator - -### Test Helpers -- `create_test_service()` - Service initialization with mocks -- `generate_sample_market_data()` - Realistic market data -- `generate_sample_news_events()` - Sentiment-scored news - -## Coverage Analysis - -### service.rs (400 lines) Coverage Estimate - -| Section | Lines | Tests | Coverage | -|---------|-------|-------|----------| -| Validation logic | 40 | 6 | 100% | -| Start backtest | 60 | 6 | 90% | -| Get status | 20 | 2 | 100% | -| Get results | 45 | 3 | 80% | -| List backtests | 25 | 3 | 90% | -| Subscribe progress | 25 | 2 | 85% | -| Stop backtest | 30 | 3 | 90% | -| Background execution | 100 | 2 | 40% | -| Helper functions | 55 | - | 30% | - -**Estimated Coverage**: 70-75% of service.rs - -### Uncovered Areas -1. **Background execution** (lines 248-350): - - Full strategy engine execution - - Performance metric calculation - - Progress broadcasting internals - -2. **Model loading** (lines 104-210): - - Historical model version loading - - Time-based model selection - - Model cache integration - -3. **Advanced features**: - - Equity curve generation (line 522) - - Drawdown period calculation (line 523) - - Total count aggregation (line 548) - -## Test Execution - -### Prerequisites -- PostgreSQL (for repository storage) -- Mock repositories (provided in `mock_repositories.rs`) -- Tokio async runtime - -### Running Tests -```bash -# Run all service tests -cargo test -p backtesting_service --test service_tests - -# Run specific test -cargo test -p backtesting_service test_start_backtest_success - -# Run with output -cargo test -p backtesting_service --test service_tests -- --nocapture -``` - -### Test Features -- **Async execution**: All tests use `#[tokio::test]` -- **Isolation**: Each test creates fresh service instance -- **Concurrency**: Tests validate parallel backtest execution -- **Error handling**: All error paths explicitly tested - -## Quality Standards Met - -โœ… **Mock gRPC requests/responses** - tonic::Request/Response used -โœ… **Test all error paths** - 4/4 tonic::Status codes tested -โœ… **Validate protobuf serialization** - Request/response conversion verified -โœ… **Concurrent backtest isolation** - 2 concurrency tests -โœ… **No workarounds** - Real mock implementations, no stubs -โœ… **Edge cases** - Invalid inputs, resource limits, not found scenarios - -## Integration with Existing Tests - -### Existing Test Files -- `integration_tests.rs` - High-level integration (minimal) -- `strategy_execution.rs` - Strategy engine tests -- `performance_metrics.rs` - Performance calculation tests -- `data_replay.rs` - Market data replay tests -- `report_generation.rs` - Report generation tests -- `mock_repositories.rs` - Mock infrastructure - -### Total Test Suite -- **Existing tests**: 74 async + 41 sync = 115 tests -- **New tests**: 22 async tests -- **Total**: 137 tests for backtesting service - -## Expected Impact - -### Coverage Improvement -- **Before**: ~45% service coverage (estimated) -- **After**: ~70-75% service coverage -- **Gain**: +25-30% coverage on service.rs - -### Test Confidence -- โœ… All 6 gRPC RPCs tested -- โœ… All validation paths covered -- โœ… All error codes verified -- โœ… Concurrent operations validated -- โœ… Full workflow integration tested - -## Next Steps (Optional Enhancements) - -1. **Background execution tests** (10-15 tests): - - Mock strategy engine execution - - Progress event streaming validation - - Performance metric calculation edge cases - -2. **Model loading tests** (5-8 tests): - - Version-specific model loading - - Time-based model selection - - Cache miss scenarios - -3. **Stream integration tests** (3-5 tests): - - Progress event sequence validation - - Stream error handling - - Client disconnect handling - -**Estimated effort**: 2-3 hours for complete 100% coverage - ---- - -**Report Generated**: 2025-10-06 -**Agent**: Wave 113 Agent 6 -**Status**: โœ… COMPLETE - 22 tests, 70-75% coverage, no workarounds diff --git a/services/backtesting_service/tests/fixtures/ARCHITECTURE.md b/services/backtesting_service/tests/fixtures/ARCHITECTURE.md deleted file mode 100644 index 53d828f26..000000000 --- a/services/backtesting_service/tests/fixtures/ARCHITECTURE.md +++ /dev/null @@ -1,466 +0,0 @@ -# Test Fixtures Architecture - -## System Overview - -``` -โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” -โ”‚ Test Suite Layer โ”‚ -โ”‚ (strategy_tests.rs, performance_tests.rs, integration_tests.rs) โ”‚ -โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ - โ”‚ - โ”‚ imports - โ–ผ -โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” -โ”‚ Fixtures & Helpers API โ”‚ -โ”‚ โ”‚ -โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ -โ”‚ โ”‚ Fixtures Module โ”‚ โ”‚ Helpers Module โ”‚ โ”‚ -โ”‚ โ”‚ (cached loading) โ”‚ โ”‚ (validation utilities) โ”‚ โ”‚ -โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ -โ”‚ โ”‚ โ€ข get_es_fut_bars() โ”‚ โ”‚ โ€ข assert_valid_ohlcv() โ”‚ โ”‚ -โ”‚ โ”‚ โ€ข get_nq_fut_bars() โ”‚ โ”‚ โ€ข assert_chronological() โ”‚ โ”‚ -โ”‚ โ”‚ โ€ข get_cl_fut_bars() โ”‚ โ”‚ โ€ข assert_price_range() โ”‚ โ”‚ -โ”‚ โ”‚ โ€ข get_regime_sample()โ”‚ โ”‚ โ€ข assert_valid_trade() โ”‚ โ”‚ -โ”‚ โ”‚ โ€ข get_multi_symbol() โ”‚ โ”‚ โ€ข calculate_volatility() โ”‚ โ”‚ -โ”‚ โ”‚ โ”‚ โ”‚ โ€ข generate_quality_report()โ”‚ โ”‚ -โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ -โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ - โ”‚ - โ”‚ loads (first call) - โ–ผ -โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” -โ”‚ Static Cache Layer โ”‚ -โ”‚ (once_cell::sync::Lazy) โ”‚ -โ”‚ โ”‚ -โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ -โ”‚ โ”‚ ES_FUT_CACHE โ”‚ โ”‚ NQ_FUT_CACHE โ”‚ โ”‚ CL_FUT_CACHE โ”‚ โ”‚ -โ”‚ โ”‚ ~50KB โ”‚ โ”‚ ~50KB โ”‚ โ”‚ ~180KB โ”‚ โ”‚ -โ”‚ โ”‚ Arc โ”‚ โ”‚ Arc โ”‚ โ”‚ Arc โ”‚ โ”‚ -โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ -โ”‚ โ–ฒ โ–ฒ โ–ฒ โ”‚ -โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ - โ”‚ โ”‚ โ”‚ - โ”‚ reads (first time only) โ”‚ - โ–ผ โ–ผ โ–ผ -โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” -โ”‚ DBN Data Source Layer โ”‚ -โ”‚ (backtesting_service::dbn_*) โ”‚ -โ”‚ โ”‚ -โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ -โ”‚ โ”‚ DbnDataSource โ”‚ โ”‚ -โ”‚ โ”‚ โ”‚ โ”‚ -โ”‚ โ”‚ โ€ข Parses DBN binary format โ”‚ โ”‚ -โ”‚ โ”‚ โ€ข Converts to MarketData structs โ”‚ โ”‚ -โ”‚ โ”‚ โ€ข Handles OHLCV aggregation โ”‚ โ”‚ -โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ -โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ - โ”‚ - โ”‚ reads - โ–ผ -โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” -โ”‚ File System Layer โ”‚ -โ”‚ โ”‚ -โ”‚ test_data/real/databento/ โ”‚ -โ”‚ โ”œโ”€โ”€ ES.FUT_ohlcv-1m_2024-01-02.dbn (95KB) โ”‚ -โ”‚ โ”œโ”€โ”€ NQ.FUT_ohlcv-1m_2024-01-02.dbn (93KB) โ”‚ -โ”‚ โ””โ”€โ”€ CL.FUT_ohlcv-1m_2024-01-02.dbn (1.5MB) โ”‚ -โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ -``` - -## Data Flow - -### Cold Cache (First Test) - -``` -Test calls get_es_fut_bars() - โ†“ -Check ES_FUT_CACHE (empty) - โ†“ -Load DBN file from disk (8ms) - โ†“ -Parse DBN โ†’ Vec - โ†“ -Store in ES_FUT_CACHE - โ†“ -Return cloned Arc (0.1ฮผs) - โ†“ -Test uses data -``` - -### Warm Cache (Subsequent Tests) - -``` -Test calls get_es_fut_bars() - โ†“ -Check ES_FUT_CACHE (hit!) - โ†“ -Return cloned Arc (0.1ฮผs) - โ†“ -Test uses data - -Total: 0.1ฮผs (100,000x faster than cold!) -``` - -## Thread Safety Model - -``` -โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” -โ”‚ Static Cache Entry โ”‚ -โ”‚ โ”‚ -โ”‚ Lazy>>>> โ”‚ -โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ -โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€โ”€ Data payload โ”‚ -โ”‚ โ”‚ โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ Mutable access control โ”‚ -โ”‚ โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ Multiple reader support โ”‚ -โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ Shared ownership โ”‚ -โ”‚ โ”‚ -โ”‚ Properties: โ”‚ -โ”‚ โ€ข Lazy: Initialized on first access โ”‚ -โ”‚ โ€ข Arc: Cheap cloning, thread-safe reference counting โ”‚ -โ”‚ โ€ข RwLock: Multiple concurrent readers โ”‚ -โ”‚ โ€ข Option: Tracks cache state (None = empty, Some = loaded)โ”‚ -โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ - -Concurrent Access Pattern: -โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” -โ”‚ Test 1 โ”‚ โ”‚ Test 2 โ”‚ โ”‚ Test 3 โ”‚ (Parallel tests) -โ””โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”˜ - โ”‚ โ”‚ โ”‚ - โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ - โ”‚ - โ–ผ - โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” - โ”‚ RwLock::read โ”‚ (All readers acquire lock) - โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ - โ”‚ - โ–ผ - โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” - โ”‚ Cached Data โ”‚ (Shared read access) - โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ - โ”‚ - โ–ผ - โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” - โ”‚ โ”‚ โ”‚ - โ–ผ โ–ผ โ–ผ -โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” -โ”‚ Clone 1 โ”‚ โ”‚ Clone 2 โ”‚ โ”‚ Clone 3 โ”‚ (Arc clones, no copy) -โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ -``` - -## Memory Layout - -``` -โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” -โ”‚ Process Memory โ”‚ -โ”‚ โ”‚ -โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ -โ”‚ โ”‚ Stack (per thread) โ”‚ โ”‚ -โ”‚ โ”‚ โ€ข Test local variables โ”‚ โ”‚ -โ”‚ โ”‚ โ€ข Function call frames โ”‚ โ”‚ -โ”‚ โ”‚ โ€ข Arc references (~8 bytes each) โ”‚ โ”‚ -โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ -โ”‚ โ”‚ -โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ -โ”‚ โ”‚ Heap (shared) โ”‚ โ”‚ -โ”‚ โ”‚ โ”‚ โ”‚ -โ”‚ โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ โ”‚ -โ”‚ โ”‚ โ”‚ ES_FUT_CACHE (static, initialized once) โ”‚ โ”‚ โ”‚ -โ”‚ โ”‚ โ”‚ Size: ~50KB (390 bars ร— 128 bytes/bar) โ”‚ โ”‚ โ”‚ -โ”‚ โ”‚ โ”‚ Location: Static memory segment โ”‚ โ”‚ โ”‚ -โ”‚ โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ โ”‚ -โ”‚ โ”‚ โ”‚ โ”‚ -โ”‚ โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ โ”‚ -โ”‚ โ”‚ โ”‚ NQ_FUT_CACHE โ”‚ โ”‚ โ”‚ -โ”‚ โ”‚ โ”‚ Size: ~50KB โ”‚ โ”‚ โ”‚ -โ”‚ โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ โ”‚ -โ”‚ โ”‚ โ”‚ โ”‚ -โ”‚ โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ โ”‚ -โ”‚ โ”‚ โ”‚ CL_FUT_CACHE โ”‚ โ”‚ โ”‚ -โ”‚ โ”‚ โ”‚ Size: ~180KB (1440 bars) โ”‚ โ”‚ โ”‚ -โ”‚ โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ โ”‚ -โ”‚ โ”‚ โ”‚ โ”‚ -โ”‚ โ”‚ Total: ~280KB (static, allocated once) โ”‚ โ”‚ -โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ -โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ -``` - -## Performance Characteristics - -### Time Complexity - -| Operation | Cold Cache | Warm Cache | Notes | -|-----------|------------|------------|-------| -| `get_es_fut_bars()` | O(n) | O(1) | n = file size | -| `get_regime_sample()` | O(nยทm) | O(nยทm) | n = bars, m = window | -| `get_multi_symbol_bars()` | O(kยทn) | O(k) | k = symbols | -| `assert_valid_ohlcv()` | - | O(n) | n = bars | -| `assert_chronological()` | - | O(n) | n = bars | - -### Space Complexity - -| Component | Memory | Notes | -|-----------|--------|-------| -| Static cache | O(nยทk) | n = bars, k = symbols | -| Arc clone | O(1) | Reference counting only | -| Test local data | O(n) | Full copy of bars | - -### Concurrency Model - -``` -Read-Write Lock (RwLock): -โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” -โ”‚ Concurrent Reads (unlimited) โ”‚ -โ”‚ โ”Œโ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ” โ”‚ -โ”‚ โ”‚ T1 โ”‚ โ”‚ T2 โ”‚ โ”‚ T3 โ”‚ โ”‚ T4 โ”‚ โ”‚ (Multiple readers) -โ”‚ โ””โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”˜ โ”‚ -โ”‚ โ†“ โ”‚ -โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ -โ”‚ โ”‚ Data โ”‚ โ”‚ -โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ -โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ - -Write Lock (initialization only): -โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” -โ”‚ Exclusive Write (once) โ”‚ -โ”‚ โ”Œโ”€โ”€โ”€โ”€โ” โ”‚ -โ”‚ โ”‚ T1 โ”‚ (First caller) โ”‚ -โ”‚ โ””โ”€โ”€โ”€โ”€โ”˜ โ”‚ -โ”‚ โ†“ โ”‚ -โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ -โ”‚ โ”‚ Data โ”‚ (Initialize) โ”‚ -โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ -โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ -``` - -## API Design Patterns - -### Pattern 1: Singleton Cache -```rust -static ES_FUT_CACHE: Lazy>>>> = - Lazy::new(|| Arc::new(RwLock::new(None))); - -// Benefits: -// โ€ข Thread-safe initialization (once_cell guarantees) -// โ€ข Lazy loading (only if used) -// โ€ข Shared ownership (Arc) -// โ€ข Concurrent reads (RwLock) -``` - -### Pattern 2: Result Chaining -```rust -pub async fn get_es_fut_bars() -> Result> { - // Check cache - { ... } - // Load if needed - { ... } - // Store and return - Ok(bars) -} - -// Benefits: -// โ€ข Explicit error handling -// โ€ข Composable with ? operator -// โ€ข Clear success/failure semantics -``` - -### Pattern 3: Builder Pattern (Future) -```rust -// Not implemented yet, but could extend: -let bars = DataFixture::new() - .symbol("ES.FUT") - .date("2024-01-02") - .regime(RegimeType::Trending) - .load() - .await?; -``` - -## Validation Architecture - -``` -โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” -โ”‚ Validation Layers โ”‚ -โ”‚ โ”‚ -โ”‚ Layer 1: OHLCV Relationships โ”‚ -โ”‚ โ”œโ”€ assert_valid_ohlcv() โ”‚ -โ”‚ โ”‚ โ”œโ”€ High >= Low โ”‚ -โ”‚ โ”‚ โ”œโ”€ High >= Open, Close โ”‚ -โ”‚ โ”‚ โ”œโ”€ Low <= Open, Close โ”‚ -โ”‚ โ”‚ โ””โ”€ All prices positive โ”‚ -โ”‚ โ”‚ โ”‚ -โ”‚ Layer 2: Time Series โ”‚ -โ”‚ โ”œโ”€ assert_chronological() โ”‚ -โ”‚ โ”‚ โ””โ”€ Timestamps strictly increasing โ”‚ -โ”‚ โ”œโ”€ assert_no_large_gaps() โ”‚ -โ”‚ โ”‚ โ””โ”€ Gaps <= max_minutes โ”‚ -โ”‚ โ”‚ โ”‚ -โ”‚ Layer 3: Statistical โ”‚ -โ”‚ โ”œโ”€ assert_price_range() โ”‚ -โ”‚ โ”‚ โ””โ”€ Symbol-specific bounds โ”‚ -โ”‚ โ”œโ”€ assert_volatility_bounds() โ”‚ -โ”‚ โ”‚ โ””โ”€ Annualized vol <= max โ”‚ -โ”‚ โ”‚ โ”‚ -โ”‚ Layer 4: Business Logic โ”‚ -โ”‚ โ”œโ”€ assert_valid_trade() โ”‚ -โ”‚ โ”‚ โ”œโ”€ Exit > Entry time โ”‚ -โ”‚ โ”‚ โ”œโ”€ Positive prices โ”‚ -โ”‚ โ”‚ โ””โ”€ Correct PnL calculation โ”‚ -โ”‚ โ””โ”€ assert_valid_trade_sequence() โ”‚ -โ”‚ โ”œโ”€ No overlapping trades โ”‚ -โ”‚ โ””โ”€ Chronological order โ”‚ -โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ -``` - -## Extension Points - -### Adding New Symbols -```rust -// 1. Add cache -static XYZ_CACHE: Lazy>>>> = ...; - -// 2. Add loader function -pub async fn get_xyz_bars() -> Result> { ... } - -// 3. Add to multi-symbol loader -match symbol.as_str() { - "ES.FUT" => get_es_fut_bars().await, - "NQ.FUT" => get_nq_fut_bars().await, - "XYZ" => get_xyz_bars().await, // New! - ... -} -``` - -### Adding New Regime Types -```rust -pub enum RegimeType { - Trending, - Ranging, - Volatile, - Stable, - Crisis, // New! - Breakout, // New! -} - -// Implement detection in calculate_regime_score() -match regime_type { - RegimeType::Crisis => { - // Detect flash crashes, extreme moves - if price_drop > 5.0 { ... } - } - ... -} -``` - -### Adding New Validation Rules -```rust -pub fn assert_trade_has_fee(&trade: &BacktestTrade, min_fee: f64) { - // Custom validation - assert!(trade.fee >= min_fee, "Fee too low"); -} -``` - -## Design Decisions - -### Decision 1: Static vs Dynamic Cache -**Chosen**: Static (`once_cell::sync::Lazy`) -**Alternative**: Dynamic (HashMap in struct) -**Rationale**: -- Simpler API (no struct to manage) -- Guaranteed single initialization -- Automatic cleanup at process exit -- Zero allocation for unused symbols - -### Decision 2: RwLock vs Mutex -**Chosen**: `tokio::sync::RwLock` -**Alternative**: `tokio::sync::Mutex` -**Rationale**: -- Read-heavy workload (99% reads) -- Multiple concurrent readers -- Negligible write contention (one-time init) -- Better performance at scale - -### Decision 3: Arc Clone vs Copy -**Chosen**: Clone `Arc>` -**Alternative**: Copy `Vec` -**Rationale**: -- Cheap (reference count increment) -- Memory efficient (no data duplication) -- Thread-safe (Arc guarantees) -- Fast (O(1) vs O(n)) - -### Decision 4: Panic vs Result -**Chosen**: Panic in validation helpers -**Alternative**: Return `Result<()>` -**Rationale**: -- Tests should fail fast -- Clear error location (file + line) -- Standard Rust test pattern -- No error propagation boilerplate - -## Performance Benchmarks - -### Single-Threaded Performance -``` -Operation | Cold Cache | Warm Cache | Speedup -------------------------|------------|------------|-------- -get_es_fut_bars() | 8.2ms | 0.08ฮผs | 102,500x -get_nq_fut_bars() | 7.8ms | 0.08ฮผs | 97,500x -get_cl_fut_bars() | 12.3ms | 0.15ฮผs | 82,000x -get_multi_symbol_bars()| 28.3ms | 0.31ฮผs | 91,290x -get_regime_sample() | 8.5ms | 2.5ฮผs | 3,400x -``` - -### Multi-Threaded Performance -``` -Threads | Total Time | Per-Thread | Speedup vs Serial ---------|------------|------------|------------------ -1 | 8.2ms | 8.2ms | 1x (baseline) -2 | 8.3ms | 4.15ms | 1.97x -4 | 8.4ms | 2.1ms | 3.90x -8 | 8.6ms | 1.08ms | 7.60x -16 | 9.1ms | 0.57ms | 14.39x -``` - -**Conclusion**: Near-linear scaling for concurrent reads. - -## Future Optimizations - -### Phase 1: Compression (Est. 10x memory reduction) -```rust -use zstd::Decoder; - -static COMPRESSED_CACHE: Lazy>> = ...; // Compressed - -pub async fn get_es_fut_bars() -> Result> { - let compressed = COMPRESSED_CACHE.clone(); - let decompressed = zstd::decode_all(&compressed[..])?; - // Parse decompressed data -} -``` - -### Phase 2: Memory-Mapped Files (Est. zero copy) -```rust -use memmap2::Mmap; - -static MMAP: Lazy = ...; - -pub async fn get_es_fut_bars() -> Result> { - // Zero-copy access via memory mapping - let data = &MMAP[...]; - // Parse directly from mapped memory -} -``` - -### Phase 3: Tiered Caching (Est. 50% memory reduction) -```rust -struct TieredCache { - hot: LruCache>, // 3 most recent - warm: HashMap>, // 10 recent - cold: fn() -> Result>, // Load on demand -} -``` - ---- - -**Architecture Version**: 1.0 -**Last Updated**: 2025-10-13 -**Author**: Agent 16 - Wave 153 diff --git a/services/backtesting_service/tests/fixtures/PERFORMANCE.md b/services/backtesting_service/tests/fixtures/PERFORMANCE.md deleted file mode 100644 index f892f8c33..000000000 --- a/services/backtesting_service/tests/fixtures/PERFORMANCE.md +++ /dev/null @@ -1,383 +0,0 @@ -# Test Fixtures Performance Analysis - -## Executive Summary - -The cached test fixtures provide **50-100x performance improvement** over naive DBN file loading, reducing test execution time from ~500ms to ~10ms for 100 tests. - -## Performance Metrics - -### Without Caching (Baseline) - -| Operation | Time | Notes | -|-----------|------|-------| -| Load ES.FUT DBN file | 5-10ms | Per test | -| Load NQ.FUT DBN file | 5-10ms | Per test | -| Load CL.FUT DBN file | 10-15ms | Larger file (1.5MB) | -| **100 tests (sequential)** | **500-1000ms** | Repeated I/O overhead | - -### With Caching (This Implementation) - -| Operation | Time | Notes | -|-----------|------|-------| -| **First load (cold cache)** | 5-10ms | One-time cost | -| **Subsequent loads (warm cache)** | ~0.1ฮผs | Memory read only | -| **100 tests (sequential)** | **~10ms** | 99 cached + 1 cold | -| **Speedup** | **50-100x** | Dramatic improvement | - -## Detailed Benchmarks - -### ES.FUT (95KB, ~390 bars) - -``` -Test Run 1: - Cold cache: 8.2ms - Warm cache: 0.08ฮผs - Speedup: 102,500x - -Test Run 2: - Cold cache: 7.9ms - Warm cache: 0.09ฮผs - Speedup: 87,778x - -Test Run 3: - Cold cache: 8.5ms - Warm cache: 0.07ฮผs - Speedup: 121,429x - -Average speedup: 103,902x -``` - -### NQ.FUT (93KB, ~390 bars) - -``` -Test Run 1: - Cold cache: 7.8ms - Warm cache: 0.08ฮผs - Speedup: 97,500x - -Average speedup: ~100,000x -``` - -### CL.FUT (1.5MB, ~1440 bars) - -``` -Test Run 1: - Cold cache: 12.3ms - Warm cache: 0.15ฮผs - Speedup: 82,000x - -Average speedup: ~80,000x (larger dataset) -``` - -## Test Suite Performance Impact - -### Small Test Suite (10 tests) - -| Metric | Without Cache | With Cache | Improvement | -|--------|---------------|------------|-------------| -| Total time | 50-100ms | 10ms | 5-10x | -| Per test | 5-10ms | ~1ms | 5-10x | - -### Medium Test Suite (100 tests) - -| Metric | Without Cache | With Cache | Improvement | -|--------|---------------|------------|-------------| -| Total time | 500-1000ms | 10ms | 50-100x | -| Per test | 5-10ms | 0.1ms | 50-100x | - -### Large Test Suite (1000 tests) - -| Metric | Without Cache | With Cache | Improvement | -|--------|---------------|------------|-------------| -| Total time | 5-10 seconds | 100ms | 50-100x | -| Per test | 5-10ms | 0.1ms | 50-100x | - -## Memory Footprint - -### Cached Data Size - -| Symbol | Bars | Memory (estimated) | -|--------|------|-------------------| -| ES.FUT | ~390 | ~50KB | -| NQ.FUT | ~390 | ~50KB | -| CL.FUT | ~1440 | ~180KB | -| **Total** | **~2220** | **~280KB** | - -**Note**: Memory usage is minimal (< 300KB) for 3 symbols with full trading day data. - -## Concurrency Performance - -### Thread Safety Overhead - -The implementation uses `tokio::sync::RwLock` for thread-safe access: - -``` -Single-threaded access: 0.1ฮผs -Multi-threaded access: 0.2ฮผs -Overhead: ~0.1ฮผs (negligible) -``` - -### Concurrent Read Performance - -``` -10 concurrent reads: - Total time: 0.5ฮผs - Per-thread: 0.05ฮผs - Linear scaling: YES - -100 concurrent reads: - Total time: 5ฮผs - Per-thread: 0.05ฮผs - Linear scaling: YES -``` - -**Conclusion**: Excellent concurrency with minimal contention. - -## Comparison with Alternatives - -### Alternative 1: Load on Demand (No Cache) - -```rust -#[tokio::test] -async fn test_strategy() { - let data_source = DbnDataSource::new(file_mapping).await?; - let bars = data_source.load_ohlcv_bars("ES.FUT").await?; - // 5-10ms PER TEST -} -``` - -**Problems**: -- Repeated file I/O -- Slow test execution -- File system contention - -### Alternative 2: In-Memory Database (Redis/SQLite) - -```rust -#[tokio::test] -async fn test_strategy() { - let bars = load_from_redis("ES.FUT").await?; - // 1-2ms per test (network/disk overhead) -} -``` - -**Problems**: -- Additional infrastructure dependency -- Network/disk latency -- Serialization overhead -- More complex setup - -### Alternative 3: Static Cache (This Implementation) โœ… - -```rust -#[tokio::test] -async fn test_strategy() { - let bars = get_es_fut_bars().await?; - // 0.1ฮผs per test (memory read) -} -``` - -**Benefits**: -- **Fastest**: Memory-only access -- **Simplest**: No external dependencies -- **Thread-safe**: Concurrent access -- **Zero-cost abstraction**: Minimal overhead - -## Real-World Impact - -### CI/CD Pipeline - -**Before caching**: -``` -Test suite: 1000 tests -DBN data tests: 500 tests -DBN I/O time: 500 * 8ms = 4 seconds -Total test time: 30 seconds -``` - -**After caching**: -``` -Test suite: 1000 tests -DBN data tests: 500 tests -DBN I/O time: 8ms (first test only) -Total test time: 26 seconds (13% faster) -``` - -### Developer Workflow - -**Before caching**: -``` -Run single test: 8ms -Run test 10 times (TDD): 80ms -Perceived: Sluggish -``` - -**After caching**: -``` -Run single test: 8ms (first), 0.1ฮผs (subsequent) -Run test 10 times (TDD): 8ms total -Perceived: Instant -``` - -## Optimization Techniques - -### 1. Lazy Loading -- Data loaded only when first accessed -- Avoids loading unused symbols -- Reduces memory footprint - -### 2. Singleton Pattern -- Single data source instance -- Shared across all tests -- Eliminates duplicate loads - -### 3. RwLock vs Mutex -- `RwLock`: Multiple concurrent reads -- `Mutex`: Single access (slower) -- Choice: `RwLock` for read-heavy workload - -### 4. Arc Cloning -- Cheap reference counting -- Zero-copy data sharing -- No serialization overhead - -## Scalability - -### Adding More Symbols - -| Symbols | Memory | First Load | Subsequent | -|---------|--------|------------|------------| -| 3 | ~280KB | 25ms | 0.3ฮผs | -| 10 | ~900KB | 80ms | 1ฮผs | -| 100 | ~9MB | 800ms | 10ฮผs | - -**Conclusion**: Linear scaling, manageable for typical test suites. - -### Memory Constraints - -**Maximum practical symbols**: ~100-200 -**Maximum practical memory**: ~20-30MB -**Typical usage**: 3-10 symbols (~300KB-1MB) - -## Best Practices - -### โœ… DO - -1. **Reuse cached data across tests** - ```rust - let bars = get_es_fut_bars().await?; - ``` - -2. **Load once, use many times** - ```rust - // Setup (once) - let bars = get_es_fut_bars().await?; - - // Multiple tests - test_strategy_1(&bars); - test_strategy_2(&bars); - test_strategy_3(&bars); - ``` - -3. **Use regime samples for targeted tests** - ```rust - let trending = get_regime_sample(RegimeType::Trending).await?; - ``` - -### โŒ DON'T - -1. **Don't bypass cache** - ```rust - // SLOW (5-10ms per test) - let data_source = DbnDataSource::new(file_mapping).await?; - let bars = data_source.load_ohlcv_bars("ES.FUT").await?; - ``` - -2. **Don't generate synthetic data unnecessarily** - ```rust - // LESS REALISTIC - let bars = generate_fake_bars(100); - ``` - -3. **Don't load in test setup hooks** - ```rust - // ANTI-PATTERN (repeated loading) - #[before_each] - fn setup() { - load_dbn_file(); // Called for EACH test - } - ``` - -## Monitoring and Profiling - -### Measure Cache Hit Rate - -```rust -static CACHE_HITS: AtomicU64 = AtomicU64::new(0); -static CACHE_MISSES: AtomicU64 = AtomicU64::new(0); - -// Track in get_es_fut_bars() -if cache.is_some() { - CACHE_HITS.fetch_add(1, Ordering::Relaxed); -} else { - CACHE_MISSES.fetch_add(1, Ordering::Relaxed); -} - -// Report at end of test run -println!("Cache hit rate: {:.1}%", - hits as f64 / (hits + misses) as f64 * 100.0); -``` - -**Expected hit rate**: >95% for typical test suites - -### Profile with `cargo flamegraph` - -```bash -cargo flamegraph --test fixtures_tests -- test_performance_comparison -``` - -**Expected profile**: -- DBN I/O: <1% (first call only) -- Cache access: ~0.1% (negligible) -- Test logic: >98% (actual test work) - -## Future Optimizations - -### 1. Compressed Storage -- Store in compressed format (zstd) -- Decompress on first access -- Trade: CPU for memory (10x reduction) - -### 2. Memory-Mapped Files -- Use `memmap2` crate -- Zero-copy file access -- OS-managed paging - -### 3. Tiered Caching -- L1: Hot data (frequently accessed) -- L2: Warm data (occasionally accessed) -- L3: Cold data (load on demand) - -### 4. Pre-warming -- Load cache at test suite startup -- Parallel loading (all symbols) -- Hide latency from first test - -## Conclusion - -The cached test fixtures provide **exceptional performance** with minimal complexity: - -- โœ… **50-100x faster** than naive approach -- โœ… **0.1ฮผs** per cached access -- โœ… **Thread-safe** concurrent reads -- โœ… **<300KB** memory footprint -- โœ… **Zero** external dependencies -- โœ… **Simple** API (just call `get_es_fut_bars()`) - -**Recommendation**: Use cached fixtures for all DBN-based tests. - -## References - -- `services/backtesting_service/tests/fixtures/mod.rs` - Implementation -- `services/backtesting_service/tests/fixtures/README.md` - Usage guide -- `services/backtesting_service/tests/fixtures_tests.rs` - Benchmarks diff --git a/services/backtesting_service/tests/fixtures/QUICKSTART.md b/services/backtesting_service/tests/fixtures/QUICKSTART.md deleted file mode 100644 index cfecdc328..000000000 --- a/services/backtesting_service/tests/fixtures/QUICKSTART.md +++ /dev/null @@ -1,358 +0,0 @@ -# Test Fixtures Quick Start Guide - -## TL;DR - -**Before** (Slow, 5-10ms per test): -```rust -let mut file_mapping = HashMap::new(); -file_mapping.insert("ES.FUT".to_string(), "path/to/ES.FUT.dbn".to_string()); -let data_source = DbnDataSource::new(file_mapping).await?; -let bars = data_source.load_ohlcv_bars("ES.FUT").await?; -``` - -**After** (Fast, 0.1ฮผs per test): -```rust -use fixtures::get_es_fut_bars; - -let bars = get_es_fut_bars().await?; // That's it! -``` - ---- - -## Quick Examples - -### 1. Load Real Data (Fastest Way) -```rust -use fixtures::{get_es_fut_bars, get_nq_fut_bars, get_cl_fut_bars}; - -#[tokio::test] -async fn test_my_strategy() -> anyhow::Result<()> { - let bars = get_es_fut_bars().await?; // Cached, fast - // Test your code... - Ok(()) -} -``` - -### 2. Validate Data Quality -```rust -use helpers::{assert_valid_ohlcv, assert_chronological}; - -#[test] -fn test_data_quality() { - let bars = load_data(); - assert_valid_ohlcv(&bars); // Validates price relationships - assert_chronological(&bars); // Validates timestamp order -} -``` - -### 3. Test Specific Market Conditions -```rust -use fixtures::{get_regime_sample, RegimeType}; - -#[tokio::test] -async fn test_trending_strategy() -> anyhow::Result<()> { - let bars = get_regime_sample(RegimeType::Trending).await?; - // bars now contain trending market data - Ok(()) -} -``` - -### 4. Multi-Symbol Testing -```rust -use fixtures::get_multi_symbol_bars; - -#[tokio::test] -async fn test_portfolio() -> anyhow::Result<()> { - let symbols = vec!["ES.FUT", "NQ.FUT"]; - let data = get_multi_symbol_bars(&symbols).await?; - // data["ES.FUT"] โ†’ ES.FUT bars - // data["NQ.FUT"] โ†’ NQ.FUT bars - Ok(()) -} -``` - ---- - -## API Reference (One Page) - -### Load Data - -| Function | Returns | Use Case | -|----------|---------|----------| -| `get_es_fut_bars()` | Vec | E-mini S&P 500 (~390 bars) | -| `get_nq_fut_bars()` | Vec | E-mini NASDAQ-100 (~390 bars) | -| `get_cl_fut_bars()` | Vec | WTI Crude Oil (~1440 bars) | -| `get_multi_symbol_bars(&[symbols])` | HashMap | Multiple symbols at once | -| `get_bars_for_date(symbol, date)` | Vec | Specific date only | -| `get_regime_sample(regime_type)` | Vec | Trending/Ranging/Volatile/Stable | - -### Validate Data - -| Function | Validates | -|----------|-----------| -| `assert_valid_ohlcv(&bars)` | Highโ‰ฅLow, Open/Close within range, positive prices | -| `assert_chronological(&bars)` | Timestamps sorted ascending | -| `assert_price_range(&bars, symbol)` | Realistic price ranges (ES: 3000-6000) | -| `assert_no_large_gaps(&bars, max_min)` | No gaps > N minutes | - -### Validate Trades - -| Function | Validates | -|----------|-----------| -| `assert_valid_trade(&trade)` | Exit > Entry time, positive prices, PnL correct | -| `assert_valid_trade_sequence(&trades)` | No overlaps, chronological order | - -### Validate Metrics - -| Function | Validates | -|----------|-----------| -| `assert_sharpe_bounds(sharpe, min, max)` | Sharpe ratio realistic (-3 to 5) | -| `assert_drawdown_bounds(dd, max)` | Drawdown โ‰ค max% | -| `assert_win_rate_valid(rate)` | Win rate 0-100% | - -### Utilities - -| Function | Returns | Use Case | -|----------|---------|----------| -| `calculate_volatility(&bars)` | f64 | Annualized volatility % | -| `generate_quality_report(&bars)` | String | Comprehensive data analysis | - ---- - -## Regime Types - -```rust -pub enum RegimeType { - Trending, // Strong directional movement (>1.5% change) - Ranging, // Bounded oscillation (<0.8% range) - Volatile, // High fluctuations (>0.5% std dev) - Stable, // Low volatility (<0.3% std dev) -} -``` - -**Usage**: -```rust -let trending = get_regime_sample(RegimeType::Trending).await?; -let ranging = get_regime_sample(RegimeType::Ranging).await?; -let volatile = get_regime_sample(RegimeType::Volatile).await?; -let stable = get_regime_sample(RegimeType::Stable).await?; -``` - ---- - -## Performance - -| Operation | Time | Notes | -|-----------|------|-------| -| First call | 5-10ms | Load from DBN file | -| Subsequent calls | ~0.1ฮผs | Read from cache | -| 100 tests | ~10ms | 50-100x faster | - ---- - -## Common Patterns - -### Pattern 1: Basic Strategy Test -```rust -use fixtures::get_es_fut_bars; -use helpers::assert_valid_ohlcv; - -#[tokio::test] -async fn test_my_strategy() -> anyhow::Result<()> { - let bars = get_es_fut_bars().await?; - assert_valid_ohlcv(&bars); - - let signals = my_strategy.generate_signals(&bars); - assert!(!signals.is_empty()); - - Ok(()) -} -``` - -### Pattern 2: Metrics Validation -```rust -use helpers::{assert_sharpe_bounds, assert_drawdown_bounds}; - -#[test] -fn test_backtest_metrics() { - let metrics = run_backtest(); - - assert_sharpe_bounds(metrics.sharpe, -3.0, 5.0); - assert_drawdown_bounds(metrics.max_dd, 50.0); -} -``` - -### Pattern 3: Data Quality Report -```rust -use fixtures::get_es_fut_bars; -use helpers::generate_quality_report; - -#[tokio::test] -async fn test_print_report() -> anyhow::Result<()> { - let bars = get_es_fut_bars().await?; - println!("{}", generate_quality_report(&bars)); - Ok(()) -} -``` - -### Pattern 4: Multi-Regime Testing -```rust -use fixtures::{get_regime_sample, RegimeType}; - -#[tokio::test] -async fn test_all_regimes() -> anyhow::Result<()> { - for regime in &[ - RegimeType::Trending, - RegimeType::Ranging, - RegimeType::Volatile, - RegimeType::Stable, - ] { - let bars = get_regime_sample(*regime).await?; - test_strategy_on_regime(&bars, regime); - } - Ok(()) -} -``` - ---- - -## Troubleshooting - -### "File not found" error -```bash -# Check files exist -ls test_data/real/databento/*.dbn - -# Should see: -# ES.FUT_ohlcv-1m_2024-01-02.dbn -# NQ.FUT_ohlcv-1m_2024-01-02.dbn -# CL.FUT_ohlcv-1m_2024-01-02.dbn -``` - -### Tests still slow -```rust -// โŒ Don't bypass cache -let data_source = DbnDataSource::new(file_mapping).await?; - -// โœ… Use cached fixtures -let bars = get_es_fut_bars().await?; -``` - -### Need different date -```rust -// Not yet implemented (only 2024-01-02 available) -// Add more DBN files to test_data/real/databento/ -``` - ---- - -## Import Cheatsheet - -```rust -// At top of test file -mod fixtures; -mod helpers; - -// In test functions -use fixtures::{ - get_es_fut_bars, get_nq_fut_bars, get_cl_fut_bars, - get_regime_sample, RegimeType, - get_multi_symbol_bars, get_bars_for_date, -}; - -use helpers::{ - assert_valid_ohlcv, assert_chronological, assert_price_range, - assert_valid_trade, assert_valid_trade_sequence, - assert_sharpe_bounds, assert_drawdown_bounds, - calculate_volatility, generate_quality_report, -}; -``` - ---- - -## Full Example Test File - -```rust -//! My strategy tests - -mod fixtures; -mod helpers; - -use anyhow::Result; -use fixtures::{get_es_fut_bars, get_regime_sample, RegimeType}; -use helpers::{assert_valid_ohlcv, assert_chronological}; - -#[tokio::test] -async fn test_strategy_trending() -> Result<()> { - // Load trending market data (cached) - let bars = get_regime_sample(RegimeType::Trending).await?; - - // Validate data quality - assert_valid_ohlcv(&bars); - assert_chronological(&bars); - - // Test strategy - let signals = my_trend_strategy.generate(&bars); - assert!(!signals.is_empty(), "Should generate signals in trend"); - - Ok(()) -} - -#[tokio::test] -async fn test_strategy_ranging() -> Result<()> { - // Load ranging market data - let bars = get_regime_sample(RegimeType::Ranging).await?; - - // Validate - assert_valid_ohlcv(&bars); - - // Test - let signals = my_mean_reversion_strategy.generate(&bars); - assert!(!signals.is_empty(), "Should generate signals in range"); - - Ok(()) -} - -#[tokio::test] -async fn test_full_day() -> Result<()> { - // Load full day of real data - let bars = get_es_fut_bars().await?; - - // Comprehensive validation - assert_valid_ohlcv(&bars); - assert_chronological(&bars); - - // Run full backtest - let results = backtest(&bars); - - // Validate metrics - use helpers::{assert_sharpe_bounds, assert_drawdown_bounds}; - assert_sharpe_bounds(results.sharpe, -3.0, 5.0); - assert_drawdown_bounds(results.max_dd, 50.0); - - Ok(()) -} -``` - ---- - -## Next Steps - -1. **Copy pattern above** for your tests -2. **Replace manual DBN loading** with `get_es_fut_bars()` -3. **Add validation helpers** to catch bugs early -4. **Run tests** and enjoy 50-100x speedup! - ---- - -## More Info - -- Full documentation: `fixtures/README.md` -- Performance analysis: `fixtures/PERFORMANCE.md` -- Examples: `fixtures_tests.rs` - ---- - -**Questions?** See comprehensive docs in `fixtures/README.md` - -**Quick start**: Just call `get_es_fut_bars().await?` and you're done! diff --git a/services/backtesting_service/tests/fixtures/README.md b/services/backtesting_service/tests/fixtures/README.md deleted file mode 100644 index 70903432e..000000000 --- a/services/backtesting_service/tests/fixtures/README.md +++ /dev/null @@ -1,431 +0,0 @@ -# Test Fixtures and Helpers Documentation - -## Overview - -This module provides **high-performance, cached access** to real DBN market data for backtesting tests. All data is loaded once and cached in static memory, dramatically improving test execution speed. - -## Architecture - -``` -tests/ -โ”œโ”€โ”€ fixtures/ -โ”‚ โ””โ”€โ”€ mod.rs # Cached data loading (singleton pattern) -โ””โ”€โ”€ helpers.rs # Validation utilities -``` - -## Performance Characteristics - -### Without Caching (Naive Approach) -- **Per test**: 5-10ms (DBN file I/O) -- **100 tests**: 500-1000ms total -- **Problem**: Repeated file reads, slow test suites - -### With Caching (This Module) -- **First test**: 5-10ms (one-time load) -- **Subsequent tests**: ~0.1ฮผs (memory read) -- **100 tests**: ~10ms total after first load -- **Benefit**: **50-100x faster** test execution - -## Usage Guide - -### 1. Basic Data Loading - -```rust -use fixtures::{get_es_fut_bars, get_nq_fut_bars, get_cl_fut_bars}; - -#[tokio::test] -async fn test_with_real_data() -> anyhow::Result<()> { - // Load ES.FUT data (cached after first call) - let bars = get_es_fut_bars().await?; - - assert!(!bars.is_empty()); - assert_eq!(bars[0].symbol, "ES.FUT"); - assert!(bars.len() > 350); // ~390 bars for trading day - - Ok(()) -} -``` - -### 2. Multi-Symbol Testing - -```rust -use fixtures::get_multi_symbol_bars; - -#[tokio::test] -async fn test_multiple_symbols() -> anyhow::Result<()> { - // Load multiple symbols in parallel - let symbols = vec!["ES.FUT", "NQ.FUT", "CL.FUT"]; - let data = get_multi_symbol_bars(&symbols).await?; - - assert_eq!(data.len(), 3); - assert!(data.contains_key("ES.FUT")); - - // Access data for each symbol - let es_bars = &data["ES.FUT"]; - let nq_bars = &data["NQ.FUT"]; - - Ok(()) -} -``` - -### 3. Regime-Specific Testing - -```rust -use fixtures::{get_regime_sample, RegimeType}; - -#[tokio::test] -async fn test_trending_strategy() -> anyhow::Result<()> { - // Get sample of trending market data - let bars = get_regime_sample(RegimeType::Trending).await?; - - // Test trend-following strategy - let signals = my_strategy.generate_signals(&bars); - assert!(signals.len() > 0); - - Ok(()) -} - -#[tokio::test] -async fn test_ranging_strategy() -> anyhow::Result<()> { - // Get sample of ranging/sideways market - let bars = get_regime_sample(RegimeType::Ranging).await?; - - // Test mean-reversion strategy - let signals = my_strategy.generate_signals(&bars); - - Ok(()) -} -``` - -### 4. Date-Specific Data - -```rust -use fixtures::get_bars_for_date; -use chrono::NaiveDate; - -#[tokio::test] -async fn test_specific_date() -> anyhow::Result<()> { - let date = NaiveDate::from_ymd_opt(2024, 1, 2) - .unwrap() - .and_hms_opt(0, 0, 0) - .unwrap() - .and_utc(); - - let bars = get_bars_for_date("ES.FUT", date).await?; - - // All bars from 2024-01-02 - for bar in &bars { - assert_eq!(bar.timestamp.date_naive(), date.date_naive()); - } - - Ok(()) -} -``` - -## Data Validation Helpers - -### 5. OHLCV Validation - -```rust -use helpers::assert_valid_ohlcv; - -#[test] -fn test_data_quality() { - let bars = load_my_data(); - - // Validates: - // - High >= Low - // - High >= Open, Close - // - Low <= Open, Close - // - All prices > 0 - // - Volume >= 0 - assert_valid_ohlcv(&bars); -} -``` - -### 6. Time Series Validation - -```rust -use helpers::{assert_chronological, assert_no_large_gaps}; - -#[test] -fn test_timestamp_quality() { - let bars = load_my_data(); - - // Ensure bars sorted by timestamp - assert_chronological(&bars); - - // Ensure no gaps > 5 minutes (for 1-minute data) - assert_no_large_gaps(&bars, 5); -} -``` - -### 7. Price Range Validation - -```rust -use helpers::assert_price_range; - -#[test] -fn test_realistic_prices() { - let bars = load_my_data(); - - // Validates prices within realistic range: - // ES.FUT: 3000-6000 - // NQ.FUT: 12000-20000 - // CL.FUT: 50-100 - assert_price_range(&bars, "ES.FUT"); -} -``` - -### 8. Trade Validation - -```rust -use helpers::{assert_valid_trade, assert_valid_trade_sequence}; - -#[test] -fn test_backtest_trades() { - let trades = run_backtest(); - - // Validate individual trade - assert_valid_trade(&trades[0]); - - // Validate entire sequence: - // - No overlapping trades - // - Chronological order - // - Valid PnL calculations - assert_valid_trade_sequence(&trades); -} -``` - -### 9. Performance Metrics Validation - -```rust -use helpers::{assert_sharpe_bounds, assert_drawdown_bounds, assert_win_rate_valid}; - -#[test] -fn test_metrics_realistic() { - let metrics = calculate_performance_metrics(); - - // Sharpe ratio between -3.0 and 5.0 - assert_sharpe_bounds(metrics.sharpe_ratio, -3.0, 5.0); - - // Max drawdown <= 50% - assert_drawdown_bounds(metrics.max_drawdown, 50.0); - - // Win rate between 0% and 100% - assert_win_rate_valid(metrics.win_rate); -} -``` - -### 10. Data Quality Reports - -```rust -use helpers::generate_quality_report; - -#[test] -fn test_print_quality_report() { - let bars = load_my_data(); - - let report = generate_quality_report(&bars); - println!("{}", report); - - // Output: - // === Data Quality Report === - // - // Total bars: 390 - // Symbol: ES.FUT - // Date range: 2024-01-02 00:00:00 to 2024-01-02 23:59:00 - // - // Price Statistics: - // Min: 4705.25 - // Max: 4748.75 - // Avg: 4725.50 - // Range: 0.92% - // - // Volatility: - // Annualized: 18.5% - // - // Quality Checks: - // OHLCV errors: 0 - // Chronology errors: 0 - // - // === End Report === -} -``` - -## Available Data - -### ES.FUT (E-mini S&P 500) -- **File**: `ES.FUT_ohlcv-1m_2024-01-02.dbn` -- **Bars**: ~390 (trading day) -- **Size**: 95KB -- **Price range**: 4700-4750 (typical 2024) -- **Use case**: General strategy testing, high liquidity - -### NQ.FUT (E-mini NASDAQ-100) -- **File**: `NQ.FUT_ohlcv-1m_2024-01-02.dbn` -- **Bars**: ~390 -- **Size**: 93KB -- **Price range**: 16500-16700 (typical 2024) -- **Use case**: Tech sector strategies - -### CL.FUT (WTI Crude Oil) -- **File**: `CL.FUT_ohlcv-1m_2024-01-02.dbn` -- **Bars**: ~1440 (24-hour trading) -- **Size**: 1.5MB -- **Price range**: 71-73 USD/barrel -- **Use case**: Extended hours trading, energy strategies - -## Regime Types - -### RegimeType::Trending -- **Characteristics**: Strong directional movement -- **Detection**: Price change > 1.5% over 60 bars -- **Use case**: Trend-following strategies - -### RegimeType::Ranging -- **Characteristics**: Bounded oscillation, sideways movement -- **Detection**: Price range < 0.8% over 60 bars -- **Use case**: Mean-reversion strategies - -### RegimeType::Volatile -- **Characteristics**: High price fluctuations -- **Detection**: Standard deviation > 0.5% -- **Use case**: Options strategies, volatility trading - -### RegimeType::Stable -- **Characteristics**: Low volatility, steady prices -- **Detection**: Standard deviation < 0.3% -- **Use case**: Low-risk strategies, carry trades - -## Performance Tips - -### โœ… DO - -```rust -// Reuse cached data across tests -let bars = get_es_fut_bars().await?; - -// Use regime samples for targeted testing -let trending = get_regime_sample(RegimeType::Trending).await?; - -// Validate data quality with helpers -assert_valid_ohlcv(&bars); -assert_chronological(&bars); -``` - -### โŒ DON'T - -```rust -// Don't load DBN files directly (bypass cache) -let data_source = DbnDataSource::new(file_mapping).await?; // SLOW - -// Don't generate synthetic data when real data available -let fake_bars = generate_fake_bars(100); // LESS REALISTIC - -// Don't skip validation (catch bugs early) -// (missing assert_valid_ohlcv check) -``` - -## Testing Best Practices - -### 1. Fast Unit Tests -```rust -// Use small samples for fast iteration -let bars = get_sample_real_data(50).await?; -``` - -### 2. Comprehensive Integration Tests -```rust -// Use full datasets for thorough testing -let bars = get_es_fut_bars().await?; -``` - -### 3. Regime-Specific Tests -```rust -// Test each regime type separately -let trending = get_regime_sample(RegimeType::Trending).await?; -let ranging = get_regime_sample(RegimeType::Ranging).await?; -``` - -### 4. Multi-Symbol Tests -```rust -// Test portfolio strategies with multiple symbols -let data = get_multi_symbol_bars(&["ES.FUT", "NQ.FUT"]).await?; -``` - -## Thread Safety - -All fixtures use `once_cell::sync::Lazy` and `tokio::sync::RwLock` for safe concurrent access: - -```rust -// Multiple tests can access cache concurrently -#[tokio::test] -async fn test_1() { - let bars = get_es_fut_bars().await?; // Thread-safe -} - -#[tokio::test] -async fn test_2() { - let bars = get_es_fut_bars().await?; // Same cache, different test -} -``` - -## Migration Guide - -### Before (Slow, No Cache) -```rust -#[tokio::test] -async fn test_old_way() { - let mut file_mapping = HashMap::new(); - file_mapping.insert( - "ES.FUT".to_string(), - "../../test_data/real/databento/ES.FUT_ohlcv-1m_2024-01-02.dbn".to_string() - ); - let data_source = DbnDataSource::new(file_mapping).await?; - let bars = data_source.load_ohlcv_bars("ES.FUT").await?; - // 5-10ms PER TEST -} -``` - -### After (Fast, Cached) -```rust -#[tokio::test] -async fn test_new_way() { - let bars = get_es_fut_bars().await?; - // ~0.1ฮผs after first load -} -``` - -## Troubleshooting - -### Cache Not Working? -- Ensure using `get_es_fut_bars()` not direct DBN loading -- Check `CACHED_ES_BARS` static is initialized - -### File Not Found? -- Verify DBN files exist: `ls test_data/real/databento/` -- Check `get_project_root()` finds correct directory - -### Test Failures? -- Use `generate_quality_report()` to inspect data -- Check validation helpers for detailed error messages - -## Examples Directory - -See `services/backtesting_service/tests/` for full examples: -- `dbn_integration_tests.rs` - DBN loading patterns -- `strategy_execution.rs` - Strategy testing with real data -- `performance_metrics.rs` - Metrics validation - -## Contributing - -When adding new fixtures: -1. Add static cache variable -2. Implement cached loading function -3. Add unit tests -4. Update this documentation - -## License - -Part of Foxhunt HFT Trading System - See project LICENSE diff --git a/services/broker_gateway_service/DEPLOYMENT.md b/services/broker_gateway_service/DEPLOYMENT.md deleted file mode 100644 index 7ef0c616b..000000000 --- a/services/broker_gateway_service/DEPLOYMENT.md +++ /dev/null @@ -1,309 +0,0 @@ -# Broker Gateway Service - Deployment Guide - -## Overview - -This directory contains production-ready deployment configurations for the Foxhunt Broker Gateway Service. The service routes orders to AMP Futures broker via FIX protocol (MVP mode: database persistence only, FIX protocol deferred to Phase 2). - -## Deployment Files - -### Docker - -- `Dockerfile.production` - Multi-stage production build (cargo-chef caching, GLIBC 2.35) -- `docker-compose.yml` - Local development stack (service + PostgreSQL + Redis + Prometheus + Grafana) -- `prometheus.yml` - Prometheus scrape configuration - -### Kubernetes - -- `k8s/deployment.yaml` - StatefulSet with 2 replicas, health probes, security context -- `k8s/service.yaml` - ClusterIP + LoadBalancer services -- `k8s/configmap.yaml` - Environment variables and configuration -- `k8s/secret.yaml` - CQG credentials (replace with actual values) -- `k8s/hpa.yaml` - Horizontal Pod Autoscaler (CPU 70%, 2-10 pods) - -### Grafana - -- `grafana/provisioning/datasources/prometheus.yml` - Prometheus datasource -- `grafana/provisioning/dashboards/dashboards.yml` - Dashboard provisioning -- `grafana/dashboards/broker_gateway_overview.json` - Pre-configured dashboard - -### Scripts - -- `../../scripts/deploy_broker_gateway.sh` - Automated deployment script with smoke tests - -## Quick Start - -### Local Development (Docker Compose) - -```bash -# Start full stack (broker + postgres + redis + prometheus + grafana) -cd services/broker_gateway_service -docker-compose up -d - -# View logs -docker-compose logs -f broker_gateway - -# Run smoke tests -curl http://localhost:8086/health -grpc_health_probe -addr=localhost:50056 - -# Access Grafana -open http://localhost:3000 # admin/foxhunt123 - -# Cleanup -docker-compose down -v -``` - -### Production Deployment (Kubernetes) - -```bash -# Build and deploy (full pipeline) -./scripts/deploy_broker_gateway.sh - -# Custom deployment options -./scripts/deploy_broker_gateway.sh \ - --namespace production \ - --image-tag v1.0.0 \ - --timeout 600 - -# Dry-run (no changes) -./scripts/deploy_broker_gateway.sh --dry-run - -# Skip build (use existing image) -./scripts/deploy_broker_gateway.sh --skip-build - -# Manual deployment -kubectl apply -f services/broker_gateway_service/k8s/ -``` - -## Configuration - -### Environment Variables - -See `k8s/configmap.yaml` for full configuration options. Key variables: - -- `DATABASE_URL` - PostgreSQL connection string -- `REDIS_URL` - Redis connection string -- `CQG_HOST` - CQG FIX server host (default: fix.amp.cqg.com) -- `CQG_PORT` - CQG FIX server port (default: 6100) -- `ENABLE_FIX_PROTOCOL` - Enable FIX protocol (default: false, MVP mode) - -### Secrets - -Replace placeholder values in `k8s/secret.yaml` with actual credentials: - -```bash -# Set environment variables -export CQG_USERNAME="your_username" -export CQG_PASSWORD="your_password" -export CQG_SENDER_COMP_ID="FOXHUNT_PROD" - -# Apply secret -envsubst < k8s/secret.yaml | kubectl apply -f - -``` - -For production, use Sealed Secrets or HashiCorp Vault for secret management. - -## Monitoring - -### Health Checks - -- HTTP: `http://localhost:8086/health` -- gRPC: `grpc_health_probe -addr=localhost:50056` -- Metrics: `http://localhost:9096/metrics` - -### Grafana Dashboards - -Access Grafana at `http://localhost:3000` (docker-compose) with credentials: -- Username: `admin` -- Password: `foxhunt123` - -Pre-configured dashboards: -- Order submission rate -- Order fill rate -- FIX message latency (P50, P95, P99) -- Database connection pool - -### Prometheus Metrics - -Key metrics exposed at `:9096/metrics`: - -- `broker_gateway_orders_submitted_total` - Total orders submitted -- `broker_gateway_orders_filled_total` - Total orders filled -- `broker_gateway_fix_latency` - FIX message latency histogram -- `broker_gateway_db_pool_connections` - Active database connections -- `broker_gateway_db_pool_idle_connections` - Idle database connections - -## Scaling - -### Horizontal Pod Autoscaler (HPA) - -Automatic scaling based on CPU/memory utilization: - -```bash -# View HPA status -kubectl get hpa -n foxhunt - -# Manual scaling -kubectl scale statefulset broker-gateway-service --replicas=5 -n foxhunt -``` - -Configuration (see `k8s/hpa.yaml`): -- Min replicas: 2 -- Max replicas: 10 -- CPU target: 70% -- Memory target: 80% - -## Troubleshooting - -### View Pod Logs - -```bash -# All pods -kubectl logs -l app=broker-gateway-service -n foxhunt --tail=100 -f - -# Specific pod -kubectl logs broker-gateway-service-0 -n foxhunt --tail=100 -f -``` - -### Check Pod Status - -```bash -kubectl get pods -l app=broker-gateway-service -n foxhunt -kubectl describe pod broker-gateway-service-0 -n foxhunt -``` - -### Exec into Pod - -```bash -kubectl exec -it broker-gateway-service-0 -n foxhunt -- /bin/bash -``` - -### Rollback Deployment - -```bash -# View rollout history -kubectl rollout history statefulset/broker-gateway-service -n foxhunt - -# Rollback to previous version -kubectl rollout undo statefulset/broker-gateway-service -n foxhunt - -# Rollback to specific revision -kubectl rollout undo statefulset/broker-gateway-service --to-revision=2 -n foxhunt -``` - -## Performance Tuning - -### Resource Limits - -Default configuration (see `k8s/deployment.yaml`): - -```yaml -resources: - limits: - cpu: 2000m - memory: 512Mi - requests: - cpu: 500m - memory: 256Mi -``` - -### Database Connection Pool - -Adjust in `k8s/configmap.yaml`: - -```yaml -DATABASE_POOL_SIZE: "20" -DATABASE_MAX_CONNECTIONS: "50" -DATABASE_IDLE_TIMEOUT: "600" -``` - -### gRPC Configuration - -Tune gRPC parameters in `k8s/configmap.yaml`: - -```yaml -GRPC_MAX_CONCURRENT_STREAMS: "1000" -GRPC_KEEPALIVE_TIME: "60" -GRPC_KEEPALIVE_TIMEOUT: "20" -``` - -## Security - -### Production Checklist - -- [ ] Replace default secrets in `k8s/secret.yaml` -- [ ] Enable TLS for gRPC (`ENABLE_TLS=true`) -- [ ] Configure network policies (firewall rules) -- [ ] Enable pod security policies -- [ ] Set up RBAC roles and service accounts -- [ ] Use Sealed Secrets or Vault for secret management -- [ ] Enable audit logging -- [ ] Configure SSL for PostgreSQL connection -- [ ] Enable Redis authentication -- [ ] Review and adjust resource limits - -### Network Policies - -Create network policies to restrict pod-to-pod communication: - -```yaml -apiVersion: networking.k8s.io/v1 -kind: NetworkPolicy -metadata: - name: broker-gateway-network-policy -spec: - podSelector: - matchLabels: - app: broker-gateway-service - policyTypes: - - Ingress - - Egress - ingress: - - from: - - podSelector: - matchLabels: - app: trading-agent-service - egress: - - to: - - podSelector: - matchLabels: - app: postgres - - to: - - podSelector: - matchLabels: - app: redis -``` - -## CI/CD Integration - -### GitLab CI/CD - -Add to `.gitlab-ci.yml`: - -```yaml -deploy-broker-gateway: - stage: deploy - script: - - ./scripts/deploy_broker_gateway.sh --namespace production - only: - - main - when: manual -``` - -### GitHub Actions - -```yaml -- name: Deploy Broker Gateway - run: | - ./scripts/deploy_broker_gateway.sh --namespace production - env: - KUBECONFIG: ${{ secrets.KUBECONFIG }} -``` - -## Support - -For issues or questions: -1. Check logs: `kubectl logs -l app=broker-gateway-service -n foxhunt` -2. Review health status: `kubectl get pods -n foxhunt` -3. Check events: `kubectl get events -n foxhunt --sort-by='.lastTimestamp'` -4. Review CLAUDE.md for system architecture diff --git a/services/broker_gateway_service/METRICS_QUICK_REF.md b/services/broker_gateway_service/METRICS_QUICK_REF.md deleted file mode 100644 index 7a8ab6b7b..000000000 --- a/services/broker_gateway_service/METRICS_QUICK_REF.md +++ /dev/null @@ -1,231 +0,0 @@ -# Broker Gateway Service - Metrics Quick Reference - -**Last Updated**: 2025-11-09 -**Status**: โœ… PRODUCTION READY -**Test Coverage**: 18/18 metrics tests passing (100%) - ---- - -## Metrics Endpoint - -**URL**: `http://localhost:9096/metrics` -**Format**: Prometheus text format - -```bash -# Check metrics endpoint -curl http://localhost:9096/metrics - -# Count total metrics -curl -s http://localhost:9096/metrics | grep "^broker_gateway_" | wc -l -# Expected: 23 metrics -``` - ---- - -## Key Metrics at a Glance - -### Order Flow -```promql -# Orders per second -rate(broker_gateway_orders_submitted_total[1m]) - -# Order latency (P95) -histogram_quantile(0.95, rate(broker_gateway_order_latency_seconds_bucket[5m])) * 1000 - -# Fill rate -rate(broker_gateway_orders_filled_total[1m]) / rate(broker_gateway_orders_submitted_total[1m]) - -# Rejection rate -rate(broker_gateway_orders_rejected_total[1m]) / rate(broker_gateway_orders_submitted_total[1m]) -``` - -### FIX Session Health -```promql -# Connection status (0=disconnected, 1=connected, 2=reconnecting) -broker_gateway_fix_session_status - -# Heartbeat RTT -broker_gateway_fix_heartbeat_rtt_ms - -# Sequence gaps (should be 0) -rate(broker_gateway_sequence_number_gap_total[5m]) -``` - -### Position Tracking -```promql -# Total position value -sum(broker_gateway_position_value_usd) - -# Unrealized PnL -sum(broker_gateway_unrealized_pnl_usd) - -# Margin utilization (%) -sum(broker_gateway_margin_used_usd) / sum(broker_gateway_cash_balance_usd) * 100 -``` - -### Error Monitoring -```promql -# Error rate (%) -rate(broker_gateway_error_total[5m]) / rate(broker_gateway_orders_submitted_total[5m]) * 100 - -# Errors by severity -sum by(severity) (rate(broker_gateway_error_total[5m])) - -# Database errors -rate(broker_gateway_db_errors_total[5m]) -``` - ---- - -## Alert Summary - -| Alert | Severity | Threshold | Duration | -|-------|----------|-----------|----------| -| FIX Session Disconnected | CRITICAL | status = 0 | 60s | -| Position Mismatch | CRITICAL | > $10K | 2m | -| High Order Latency | WARNING | P95 > 100ms | 5m | -| High Error Rate | WARNING | > 5% | 5m | -| No Order Activity | WARNING | > 10m (market hours) | 1m | -| Sequence Gap | WARNING | > 0 | 1m | -| High Heartbeat RTT | WARNING | > 50ms | 5m | -| High Rejection Rate | WARNING | > 10% | 5m | -| Slow DB Queries | WARNING | P95 > 100ms | 5m | -| High Margin Usage | WARNING | > 80% | 5m | - ---- - -## Grafana Panels - -1. **Order Submission Rate**: Real-time order flow volume -2. **Order Latency (P50/P95/P99)**: Performance percentiles -3. **FIX Session Status**: Connection health (green/yellow/red gauge) -4. **Position Value Trend**: Position value over time -5. **Error Rate by Type**: Stacked bar chart by severity -6. **Top 10 Slowest Operations**: Performance bottleneck table - -**Dashboard UID**: `broker_gateway_dashboard` -**Import**: Upload `grafana/dashboard.json` - ---- - -## Common Queries - -### Health Check -```promql -# Service is healthy if: -broker_gateway_fix_session_status == 1 # FIX connected -AND rate(broker_gateway_error_total[5m]) < 0.05 # Error rate < 5% -AND histogram_quantile(0.95, rate(broker_gateway_order_latency_seconds_bucket[5m])) < 0.1 # P95 < 100ms -``` - -### Capacity Planning -```promql -# Max order rate (orders/sec) -max_over_time(rate(broker_gateway_orders_submitted_total[1m])[1h:]) - -# Avg order latency (ms) -rate(broker_gateway_order_latency_seconds_sum[5m]) / rate(broker_gateway_order_latency_seconds_count[5m]) * 1000 - -# Active orders by status -broker_gateway_active_orders -``` - -### Troubleshooting -```promql -# Recent errors by type -topk(5, sum by(error_type) (increase(broker_gateway_error_total[15m]))) - -# Slowest order types -topk(3, rate(broker_gateway_order_latency_seconds_sum[5m]) / rate(broker_gateway_order_latency_seconds_count[5m])) - -# FIX message volume -rate(broker_gateway_fix_messages_sent_total[1m]) -rate(broker_gateway_fix_messages_received_total[1m]) -``` - ---- - -## Integration Examples - -### Record Order Submission -```rust -use broker_gateway_service::metrics; - -metrics::record_order_submitted("ES.FUT", "MARKET", "BUY"); -metrics::record_order_latency("MARKET", 0.025); // 25ms -``` - -### Record Order Fill -```rust -metrics::record_order_filled("ES.FUT", "LIMIT", "SELL", 0.150); // 150ms fill time -``` - -### Update Position -```rust -metrics::update_position( - "NQ.FUT", "PROD_ACCT_1", - 5.0, // quantity - 100_000.0, // value USD - 2_500.0 // unrealized PnL -); -``` - -### Update FIX Session -```rust -metrics::update_fix_session_status("FOXHUNT-CQG", 1.0); // Connected -metrics::update_heartbeat_rtt("FOXHUNT-CQG", 15.5); // 15.5ms -``` - -### Create Tracing Span -```rust -use broker_gateway_service::tracing as bg_tracing; - -let _span = bg_tracing::span_route_order("ES.FUT", "BUY", "MARKET", 10.0); -// All logs within scope are attached to span -``` - ---- - -## Files - -| File | Purpose | -|------|---------| -| `src/metrics.rs` | Metrics definitions (645 lines) | -| `src/tracing.rs` | Tracing spans (305 lines) | -| `grafana/dashboard.json` | Dashboard (826 lines) | -| `prometheus/alerts.yml` | Alert rules (378 lines) | -| `tests/metrics_integration_test.rs` | Tests (418 lines) | - ---- - -## Testing - -```bash -# Run metrics tests -cargo test -p broker_gateway_service --test metrics_integration_test - -# Expected: 18/18 PASS - -# Build service -cargo build -p broker_gateway_service --release - -# Expected: 0 warnings, 0 errors - -# Start service -./target/release/broker_gateway_service - -# Check metrics -curl http://localhost:9096/metrics | grep broker_gateway -``` - ---- - -## Next Steps - -1. Configure Prometheus scraping (port 9096) -2. Import Grafana dashboard -3. Load Prometheus alerts -4. Configure Alertmanager routing -5. Test alerts with synthetic errors - -See `MONITORING_IMPLEMENTATION.md` for detailed setup instructions. diff --git a/services/broker_gateway_service/MONITORING_IMPLEMENTATION.md b/services/broker_gateway_service/MONITORING_IMPLEMENTATION.md deleted file mode 100644 index 6a090d5af..000000000 --- a/services/broker_gateway_service/MONITORING_IMPLEMENTATION.md +++ /dev/null @@ -1,618 +0,0 @@ -# Broker Gateway Service - Monitoring & Observability Implementation - -**Status**: โœ… COMPLETE -**Date**: 2025-11-09 -**Test Results**: 18/18 metrics tests passing (100%) -**Zero Warnings**: โœ… Compilation clean - ---- - -## Executive Summary - -Implemented comprehensive monitoring and observability infrastructure for Broker Gateway Service with: -- **23 Prometheus metrics** across 5 categories -- **OpenTelemetry distributed tracing** for all RPC methods -- **Grafana dashboard** with 6 visualization panels -- **10 Prometheus alerts** (6 CRITICAL, 4 WARNING) -- **HTTP metrics endpoint** on port 9096 -- **18 integration tests** validating all metrics - ---- - -## 1. Prometheus Metrics (23 Metrics) - -### 1.1 Order Metrics (5 metrics) - -| Metric | Type | Description | Labels | -|--------|------|-------------|--------| -| `broker_gateway_orders_submitted_total` | Counter | Orders submitted to broker | symbol, order_type, side | -| `broker_gateway_orders_filled_total` | Counter | Orders successfully filled | symbol, order_type, side | -| `broker_gateway_orders_rejected_total` | Counter | Orders rejected by broker | symbol, order_type, reason | -| `broker_gateway_orders_cancelled_total` | Counter | Order cancellation requests | symbol, status | -| `broker_gateway_orders_partial_fills_total` | Counter | Partial fills by symbol | symbol | - -**Helper Functions**: -- `record_order_submitted(symbol, order_type, side)` -- `record_order_filled(symbol, order_type, side, fill_latency_seconds)` -- `record_order_rejected(symbol, order_type, reason)` -- `record_order_cancelled(symbol, status)` -- `record_partial_fill(symbol)` - -### 1.2 Latency Metrics (3 metrics) - -| Metric | Type | Description | Buckets | -|--------|------|-------------|---------| -| `broker_gateway_order_latency_seconds` | Histogram | Order routing latency | 0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1.0s | -| `broker_gateway_fix_message_latency_ms` | Histogram | FIX message processing latency | 0.1, 0.5, 1.0, 5.0, 10.0, 50.0, 100.0ms | -| `broker_gateway_order_fill_latency_seconds` | Histogram | Time from submission to fill | 0.01, 0.1, 0.5, 1.0, 5.0, 10.0, 60.0s | - -**Helper Functions**: -- `record_order_latency(order_type, latency_seconds)` - -### 1.3 Position Metrics (5 metrics) - -| Metric | Type | Description | Labels | -|--------|------|-------------|--------| -| `broker_gateway_position_value_usd` | Gauge | Current position value | symbol, account_id | -| `broker_gateway_position_quantity` | Gauge | Position quantity (positive=long, negative=short) | symbol, account_id | -| `broker_gateway_unrealized_pnl_usd` | Gauge | Mark-to-market profit/loss | symbol, account_id | -| `broker_gateway_cash_balance_usd` | Gauge | Account cash balance | account_id | -| `broker_gateway_margin_used_usd` | Gauge | Margin utilized | account_id | - -**Helper Functions**: -- `update_position(symbol, account_id, quantity, value_usd, unrealized_pnl)` -- `update_account(account_id, cash_balance, margin_used)` - -### 1.4 FIX Session Metrics (7 metrics) - -| Metric | Type | Description | Values/Labels | -|--------|------|-------------|---------------| -| `broker_gateway_fix_session_status` | Gauge | Connection status | 0=disconnected, 1=connected, 2=reconnecting | -| `broker_gateway_sequence_number_gap_total` | Counter | Sequence gaps detected | session_id | -| `broker_gateway_fix_heartbeat_rtt_ms` | Gauge | Heartbeat round-trip time | session_id | -| `broker_gateway_fix_sender_seq_num` | IntGauge | Outbound sequence number | session_id | -| `broker_gateway_fix_target_seq_num` | IntGauge | Inbound sequence number | session_id | -| `broker_gateway_fix_messages_sent_total` | Counter | FIX messages sent by type | session_id, message_type | -| `broker_gateway_fix_messages_received_total` | Counter | FIX messages received by type | session_id, message_type | - -**Helper Functions**: -- `update_fix_session_status(session_id, status)` -- `record_sequence_gap(session_id)` -- `update_heartbeat_rtt(session_id, rtt_ms)` -- `update_sequence_numbers(session_id, sender_seq, target_seq)` -- `record_fix_message_sent(session_id, message_type, latency_ms)` -- `record_fix_message_received(session_id, message_type)` - -### 1.5 Error Metrics (3 metrics) - -| Metric | Type | Description | Labels | -|--------|------|-------------|--------| -| `broker_gateway_error_total` | Counter | Errors by type and severity | error_type, severity | -| `broker_gateway_db_errors_total` | Counter | Database operation failures | operation | -| `broker_gateway_last_order_time` | Gauge | Unix timestamp of last order | - | -| `broker_gateway_active_orders` | Gauge | Active orders by status | status | - -**Helper Functions**: -- `record_error(error_type, severity)` -- `record_db_error(operation)` -- `update_active_orders(status, count)` - ---- - -## 2. OpenTelemetry Distributed Tracing - -### 2.1 RPC Method Spans (6 spans) - -| Span Name | Attributes | Purpose | -|-----------|------------|---------| -| `broker_gateway.route_order` | symbol, side, type, quantity | Order submission tracing | -| `broker_gateway.cancel_order` | client_order_id, account_id | Order cancellation tracing | -| `broker_gateway.get_account_state` | account_id | Account query tracing | -| `broker_gateway.get_positions` | account_id, symbol | Position query tracing | -| `broker_gateway.get_session_status` | session_id | FIX session status tracing | -| `broker_gateway.stream_executions` | account_id, symbol | Execution stream tracing | - -### 2.2 Operation Spans (4 spans) - -| Span Name | Attributes | Purpose | -|-----------|------------|---------| -| `database.operation` | operation, table, system | Database query tracing | -| `fix.encode` | message_type, protocol | FIX message encoding | -| `fix.network_send` | message_type, session_id, transport | FIX network transmission | -| `fix.receive` | message_type, session_id | FIX message reception | - -### 2.3 Tracing Helper Functions - -**Event Recording**: -- `record_error(error_type, error_message, severity)` - Error event with stack trace -- `record_order_submitted(client_order_id, status)` - Order submission event -- `record_order_filled(broker_order_id, filled_quantity, fill_price)` - Order fill event -- `record_order_rejected(reason)` - Order rejection event - -**Latency Recording**: -- `record_database_latency(start)` - Database query latency (warns if >100ms) -- `record_fix_network_latency(start)` - FIX network latency (warns if >10ms) -- `record_latency(start)` - General operation latency - -**State Recording**: -- `record_fix_sequence_numbers(sender_seq, target_seq)` - FIX sequence tracking -- `record_fix_heartbeat_rtt(rtt_ms)` - FIX heartbeat latency (warns if >50ms) -- `record_account_state(balance, margin_used)` - Account state snapshot - -### 2.4 Span Attributes - -**Service Attributes**: -- `service.name` = "broker_gateway_service" -- `service.version` = CARGO_PKG_VERSION - -**Order Attributes**: -- `order.symbol`, `order.side`, `order.type`, `order.quantity` -- `order.price`, `order.client_id`, `order.broker_id`, `order.status` - -**Account Attributes**: -- `account.id`, `account.balance`, `account.margin_used` - -**FIX Session Attributes**: -- `fix.session_id`, `fix.message_type`, `fix.sender_seq`, `fix.target_seq` -- `fix.heartbeat_rtt_ms`, `fix.protocol` - -**Error Attributes**: -- `error.type`, `error.message`, `error.stack_trace`, `error.severity` - -**Performance Attributes**: -- `latency.ms`, `database.query_time_ms`, `fix.network_time_ms` - ---- - -## 3. Grafana Dashboard (6 Panels) - -**Dashboard UID**: `broker_gateway_dashboard` -**Dashboard Title**: "Broker Gateway Service - Order Routing & FIX Protocol" -**Refresh**: 10 seconds -**Time Range**: Last 1 hour - -### Panel 1: Order Submission Rate (1m) -- **Type**: Time Series (Line Chart) -- **Query**: `rate(broker_gateway_orders_submitted_total[1m])` -- **Legend**: `{{symbol}} {{order_type}} {{side}}` -- **Y-Axis**: Orders/sec -- **Purpose**: Monitor order flow volume and distribution - -### Panel 2: Order Latency (P50, P95, P99) -- **Type**: Time Series (Line Chart) -- **Queries**: - - P50: `histogram_quantile(0.50, sum by(le, order_type) (rate(broker_gateway_order_latency_seconds_bucket[5m]))) * 1000` - - P95: `histogram_quantile(0.95, sum by(le, order_type) (rate(broker_gateway_order_latency_seconds_bucket[5m]))) * 1000` - - P99: `histogram_quantile(0.99, sum by(le, order_type) (rate(broker_gateway_order_latency_seconds_bucket[5m]))) * 1000` -- **Y-Axis**: Latency (ms) -- **Thresholds**: Yellow @ 50ms, Red @ 100ms -- **Purpose**: Identify performance degradation - -### Panel 3: FIX Session Status -- **Type**: Gauge -- **Query**: `broker_gateway_fix_session_status` -- **Mappings**: - - 0 = DISCONNECTED (Red) - - 1 = CONNECTED (Green) - - 2 = RECONNECTING (Yellow) -- **Purpose**: Real-time FIX connectivity status - -### Panel 4: Position Value Trend -- **Type**: Time Series (Line Chart) -- **Query**: `broker_gateway_position_value_usd` -- **Legend**: `{{symbol}} - {{account_id}}` -- **Y-Axis**: Position Value (USD) -- **Purpose**: Track position value changes over time - -### Panel 5: Error Rate by Type -- **Type**: Time Series (Stacked Bar Chart) -- **Queries**: - - CRITICAL: `rate(broker_gateway_error_total{severity="CRITICAL"}[1m])` - - ERROR: `rate(broker_gateway_error_total{severity="ERROR"}[1m])` - - WARNING: `rate(broker_gateway_error_total{severity="WARNING"}[1m])` -- **Colors**: CRITICAL=dark-red, ERROR=red, WARNING=yellow -- **Y-Axis**: Errors/min -- **Purpose**: Monitor error rates and severity distribution - -### Panel 6: Top 10 Slowest Operations -- **Type**: Table -- **Query**: -```promql -topk(10, - (sum by(order_type) (rate(broker_gateway_order_latency_seconds_sum[5m])) - / - sum by(order_type) (rate(broker_gateway_order_latency_seconds_count[5m]))) - * 1000 -) -``` -- **Columns**: Order Type, Avg Latency (ms) -- **Sort**: Descending by latency -- **Color**: Background gradient (Green < 50ms < Yellow < 100ms < Red) -- **Purpose**: Identify performance bottlenecks - ---- - -## 4. Prometheus Alerts (10 Alerts) - -### 4.1 CRITICAL Alerts (2 alerts) - -#### Alert 1: BrokerGatewayFIXSessionDisconnected -- **Severity**: CRITICAL -- **Condition**: `broker_gateway_fix_session_status == 0` -- **Duration**: 60 seconds -- **Impact**: Trading halted - no orders can be submitted -- **Actions**: - 1. Check FIX engine logs for connection errors - 2. Verify network connectivity to AMP Futures gateway - 3. Check firewall rules and VPN tunnel - 4. Verify FIX credentials and session configuration - 5. Attempt manual FIX session reconnect - -#### Alert 5: BrokerGatewayPositionMismatch -- **Severity**: CRITICAL -- **Condition**: `abs(broker_gateway_position_value_usd - trading_service_position_value_usd) > 10000` -- **Duration**: 2 minutes -- **Impact**: Data integrity - position tracking inaccuracy -- **Actions**: - 1. HALT automated trading on affected symbol/account - 2. Reconcile position with broker's actual position - 3. Review recent execution reports for missing fills - 4. Check for FIX message sequence gaps - 5. Verify database transaction integrity - -### 4.2 WARNING Alerts (8 alerts) - -#### Alert 2: BrokerGatewayHighOrderLatency -- **Severity**: WARNING -- **Condition**: `histogram_quantile(0.95, sum by(le, order_type) (rate(broker_gateway_order_latency_seconds_bucket[5m]))) * 1000 > 100` -- **Duration**: 5 minutes -- **Impact**: Performance degraded - order routing slow -- **Target**: P95 < 100ms - -#### Alert 3: BrokerGatewayHighErrorRate -- **Severity**: WARNING -- **Condition**: `(sum by(severity) (rate(broker_gateway_error_total[5m])) / (sum(rate(broker_gateway_orders_submitted_total[5m])) + 0.001)) > 0.05` -- **Duration**: 5 minutes -- **Impact**: Reliability degraded - increased order rejections -- **Target**: Error rate < 5% - -#### Alert 4: BrokerGatewayNoOrderActivity -- **Severity**: WARNING -- **Condition**: `(time() - broker_gateway_last_order_time) > 600 and (hour() >= 9 and hour() < 16)` -- **Duration**: 1 minute -- **Impact**: Trading inactive - no orders for 10+ minutes during market hours - -#### Alert 6: BrokerGatewayFIXSequenceGap -- **Severity**: WARNING -- **Condition**: `rate(broker_gateway_sequence_number_gap_total[5m]) > 0` -- **Duration**: 1 minute -- **Impact**: Message loss - potential missed execution reports - -#### Alert 7: BrokerGatewayHighFIXHeartbeatRTT -- **Severity**: WARNING -- **Condition**: `broker_gateway_fix_heartbeat_rtt_ms > 50` -- **Duration**: 5 minutes -- **Impact**: Network latency - delayed order execution -- **Target**: RTT < 50ms - -#### Alert 8: BrokerGatewayHighOrderRejectionRate -- **Severity**: WARNING -- **Condition**: `(sum by(symbol) (rate(broker_gateway_orders_rejected_total[5m])) / (sum by(symbol) (rate(broker_gateway_orders_submitted_total[5m])) + 0.001)) > 0.10` -- **Duration**: 5 minutes -- **Impact**: Order flow disrupted - high rejection rate -- **Target**: Rejection rate < 10% - -#### Alert 9: BrokerGatewaySlowDatabaseQueries -- **Severity**: WARNING -- **Condition**: `histogram_quantile(0.95, sum by(le) (rate(database_query_duration_seconds_bucket{service="broker_gateway_service"}[5m]))) > 0.100` -- **Duration**: 5 minutes -- **Impact**: Performance degraded - slow database queries -- **Target**: P95 < 100ms - -#### Alert 10: BrokerGatewayHighMarginUsage -- **Severity**: WARNING -- **Condition**: `(broker_gateway_margin_used_usd / (broker_gateway_cash_balance_usd + 0.001)) > 0.80` -- **Duration**: 5 minutes -- **Impact**: Margin pressure - risk of margin call -- **Target**: Margin usage < 80% - ---- - -## 5. Implementation Details - -### 5.1 Files Created - -| File | Lines | Purpose | -|------|-------|---------| -| `src/metrics.rs` | 645 | Prometheus metrics definitions and helper functions | -| `src/tracing.rs` | 305 | OpenTelemetry tracing spans and event recording | -| `grafana/dashboard.json` | 826 | Grafana dashboard with 6 panels | -| `prometheus/alerts.yml` | 378 | Prometheus alerting rules (10 alerts) | -| `tests/metrics_integration_test.rs` | 418 | Integration tests for all metrics | - -**Total**: 2,572 lines of monitoring infrastructure - -### 5.2 Files Modified - -| File | Changes | Purpose | -|------|---------|---------| -| `src/lib.rs` | +4 lines | Added `metrics` and `tracing` modules, `#![deny(warnings)]` | -| `src/main.rs` | +35 lines | Added metrics HTTP endpoint on port 9096 | -| `src/service.rs` | +25 lines | Integrated metrics and tracing into `route_order` RPC | - -### 5.3 Service Endpoints - -| Endpoint | Port | Purpose | -|----------|------|---------| -| gRPC Server | 50056 | Order routing API | -| Health Check | 8086 | Liveness and readiness probes | -| **Metrics** | **9096** | **Prometheus scraping endpoint** | - -### 5.4 Metrics HTTP Endpoint - -**URL**: `http://0.0.0.0:9096/metrics` -**Format**: Prometheus text format -**Content-Type**: `text/plain; version=0.0.4` - -**Example Output**: -``` -# HELP broker_gateway_orders_submitted_total Total number of orders submitted to broker by symbol, type, and side -# TYPE broker_gateway_orders_submitted_total counter -broker_gateway_orders_submitted_total{symbol="ES.FUT",order_type="MARKET",side="BUY"} 1 - -# HELP broker_gateway_order_latency_seconds Order routing latency from gRPC to broker acknowledgment in seconds -# TYPE broker_gateway_order_latency_seconds histogram -broker_gateway_order_latency_seconds_bucket{order_type="MARKET",le="0.001"} 0 -broker_gateway_order_latency_seconds_bucket{order_type="MARKET",le="0.005"} 0 -broker_gateway_order_latency_seconds_bucket{order_type="MARKET",le="0.01"} 1 -broker_gateway_order_latency_seconds_sum{order_type="MARKET"} 0.025 -broker_gateway_order_latency_seconds_count{order_type="MARKET"} 1 - -# HELP broker_gateway_fix_session_status FIX session connection status (0=disconnected, 1=connected, 2=reconnecting) -# TYPE broker_gateway_fix_session_status gauge -broker_gateway_fix_session_status{session_id="FOXHUNT-CQG"} 1 - -# HELP broker_gateway_position_value_usd Current position value in USD by symbol and account -# TYPE broker_gateway_position_value_usd gauge -broker_gateway_position_value_usd{symbol="ES.FUT",account_id="TEST_ACCOUNT"} 50000 -``` - ---- - -## 6. Testing Results - -### 6.1 Metrics Integration Tests - -**Test Suite**: `tests/metrics_integration_test.rs` -**Tests**: 18 tests -**Result**: โœ… **100% PASS (18/18)** - -| Test | Status | Coverage | -|------|--------|----------| -| `test_order_submitted_metric` | โœ… PASS | Order submission counter | -| `test_order_filled_metric` | โœ… PASS | Order fill counter + latency histogram | -| `test_order_rejected_metric` | โœ… PASS | Order rejection counter | -| `test_order_cancelled_metric` | โœ… PASS | Order cancellation counter | -| `test_partial_fill_metric` | โœ… PASS | Partial fill counter | -| `test_order_latency_metric` | โœ… PASS | Order latency histogram | -| `test_position_metrics` | โœ… PASS | Position quantity, value, unrealized PnL | -| `test_account_metrics` | โœ… PASS | Cash balance, margin used | -| `test_fix_session_status_metric` | โœ… PASS | FIX connection status gauge | -| `test_sequence_gap_metric` | โœ… PASS | Sequence gap counter | -| `test_heartbeat_rtt_metric` | โœ… PASS | Heartbeat RTT gauge | -| `test_sequence_numbers_metric` | โœ… PASS | Sender/target sequence numbers | -| `test_fix_message_sent_metric` | โœ… PASS | FIX messages sent counter + latency | -| `test_fix_message_received_metric` | โœ… PASS | FIX messages received counter | -| `test_error_metrics` | โœ… PASS | Error counter by type and severity | -| `test_database_error_metric` | โœ… PASS | Database error counter | -| `test_active_orders_metric` | โœ… PASS | Active orders gauge by status | -| `test_last_order_time_metric` | โœ… PASS | Last order timestamp gauge | - -**Execution Time**: 0.00s (all tests run in parallel) - -### 6.2 Compilation Status - -**Command**: `cargo build -p broker_gateway_service --release` -**Result**: โœ… **CLEAN (0 errors, 0 warnings)** -**Build Time**: 48.11 seconds - ---- - -## 7. Production Readiness Checklist - -- โœ… **Metrics Defined**: 23 Prometheus metrics across 5 categories -- โœ… **Tracing Implemented**: 10 OpenTelemetry spans for distributed tracing -- โœ… **Dashboard Created**: Grafana dashboard with 6 visualization panels -- โœ… **Alerts Configured**: 10 Prometheus alerts (6 CRITICAL, 4 WARNING) -- โœ… **HTTP Endpoint**: Metrics exposed on port 9096 for Prometheus scraping -- โœ… **Integration Tests**: 18/18 tests passing (100%) -- โœ… **Zero Warnings**: Compilation clean with `#![deny(warnings)]` -- โœ… **Documentation**: Complete README with usage examples and runbooks -- โœ… **Helper Functions**: 18 convenience functions for metrics recording -- โœ… **Error Handling**: Structured error tracing with stack traces - ---- - -## 8. Next Steps - -### 8.1 Prometheus Configuration - -Add scrape target to `prometheus.yml`: -```yaml -scrape_configs: - - job_name: 'broker_gateway_service' - scrape_interval: 10s - static_configs: - - targets: ['localhost:9096'] - labels: - service: 'broker_gateway_service' - environment: 'production' -``` - -### 8.2 Alertmanager Configuration - -Configure alert routing in `alertmanager.yml`: -```yaml -route: - receiver: 'default' - routes: - - match: - service: 'broker_gateway_service' - severity: 'CRITICAL' - receiver: 'pagerduty' - - match: - service: 'broker_gateway_service' - severity: 'WARNING' - receiver: 'slack' - -receivers: - - name: 'pagerduty' - pagerduty_configs: - - service_key: '' - - name: 'slack' - slack_configs: - - api_url: '' - channel: '#broker-gateway-alerts' -``` - -### 8.3 Grafana Dashboard Import - -1. Navigate to Grafana UI โ†’ Dashboards โ†’ Import -2. Upload `grafana/dashboard.json` -3. Select Prometheus datasource -4. Click "Import" - -Dashboard UID: `broker_gateway_dashboard` - -### 8.4 Load Prometheus Alerts - -Add to Prometheus configuration: -```yaml -rule_files: - - '/etc/prometheus/alerts/broker_gateway_service.yml' -``` - -Copy `prometheus/alerts.yml` to `/etc/prometheus/alerts/broker_gateway_service.yml` - ---- - -## 9. Usage Examples - -### 9.1 Record Order Submission - -```rust -use broker_gateway_service::metrics; - -// Record order submission -metrics::record_order_submitted("ES.FUT", "MARKET", "BUY"); -metrics::record_order_latency("MARKET", 0.025); // 25ms -``` - -### 9.2 Record Order Fill - -```rust -// Record order fill with latency -metrics::record_order_filled( - "ES.FUT", // symbol - "LIMIT", // order_type - "SELL", // side - 0.150 // fill_latency_seconds (150ms) -); -``` - -### 9.3 Update Position - -```rust -// Update position metrics -metrics::update_position( - "NQ.FUT", // symbol - "PROD_ACCT_1", // account_id - 5.0, // quantity (5 contracts long) - 100_000.0, // value_usd ($100K) - 2_500.0 // unrealized_pnl ($2.5K profit) -); -``` - -### 9.4 Update FIX Session Status - -```rust -// Update FIX session status -metrics::update_fix_session_status("FOXHUNT-CQG", 1.0); // Connected -metrics::update_heartbeat_rtt("FOXHUNT-CQG", 15.5); // 15.5ms RTT -metrics::update_sequence_numbers("FOXHUNT-CQG", 1234, 5678); -``` - -### 9.5 Create Tracing Span - -```rust -use broker_gateway_service::tracing as bg_tracing; - -// Create span for order routing -let _span = bg_tracing::span_route_order( - "ES.FUT", // symbol - "BUY", // side - "MARKET", // order_type - 10.0 // quantity -); - -// All log events within this scope are attached to the span -// Span automatically closes when dropped -``` - ---- - -## 10. Performance Impact - -**Metrics Collection Overhead**: Negligible (~10-50 nanoseconds per counter increment) -**Tracing Overhead**: ~1-5 microseconds per span (0.01-0.05% of 10-100ms order latency) -**Memory Usage**: ~2-5MB for Prometheus metrics registry -**HTTP Endpoint**: ~100-200 microseconds per scrape request - -**Conclusion**: Monitoring infrastructure adds <0.1% overhead to order routing latency. - ---- - -## 11. Security Considerations - -- โœ… Metrics endpoint exposed on internal port 9096 (not public-facing) -- โœ… No sensitive data exposed in metric labels (no API keys, passwords, PII) -- โœ… Account IDs and symbols are business identifiers (not sensitive) -- โœ… Tracing attributes follow OTEL semantic conventions -- โœ… Error messages sanitized (no stack dumps in metrics) - ---- - -## 12. Compliance - -**Metrics Naming**: Follows Prometheus best practices -- Counter suffixes: `_total` -- Histogram suffixes: `_seconds`, `_ms` -- Gauge suffixes: `_usd`, `_rtt_ms` -- Prefix: `broker_gateway_` for namespace isolation - -**Labels**: Follow high-cardinality best practices -- Low cardinality: symbol (10-50 values), order_type (4 values), side (2 values) -- Medium cardinality: account_id (100-1000 values) -- No unbounded cardinality (UUIDs, timestamps, etc.) - -**Tracing**: Follows OpenTelemetry semantic conventions -- Span names: `component.operation` format -- Attributes: Standard OTEL naming (e.g., `db.system`, `network.transport`) - ---- - -## 13. Conclusion - -Comprehensive monitoring and observability infrastructure successfully implemented for Broker Gateway Service with: - -โœ… **23 production-ready Prometheus metrics** -โœ… **10 distributed tracing spans with OpenTelemetry** -โœ… **6-panel Grafana dashboard** -โœ… **10 Prometheus alerts (6 CRITICAL, 4 WARNING)** -โœ… **18/18 integration tests passing** -โœ… **Zero compilation warnings** -โœ… **HTTP metrics endpoint operational** - -**Status**: ๐ŸŸข **PRODUCTION READY** - -The monitoring infrastructure provides complete visibility into order routing performance, FIX session health, position tracking, and error rates with minimal overhead (<0.1% latency impact). diff --git a/services/broker_gateway_service/docs/API.md b/services/broker_gateway_service/docs/API.md deleted file mode 100644 index a3e3018f4..000000000 --- a/services/broker_gateway_service/docs/API.md +++ /dev/null @@ -1,990 +0,0 @@ -# Broker Gateway Service API Reference - -**Version**: 0.1.0 -**Protocol**: gRPC (Protocol Buffers v3) -**Base URL**: `grpc://localhost:50056` - -## Table of Contents - -- [Overview](#overview) -- [Authentication](#authentication) -- [Service Methods](#service-methods) - - [RouteOrder](#routeorder) - - [CancelOrder](#cancelorder) - - [GetAccountState](#getaccountstate) - - [GetPositions](#getpositions) - - [GetSessionStatus](#getsessionstatus) - - [StreamExecutions](#streamexecutions) - - [HealthCheck](#healthcheck) -- [Data Types](#data-types) -- [Error Codes](#error-codes) -- [Rate Limits](#rate-limits) -- [Client Examples](#client-examples) - ---- - -## Overview - -The Broker Gateway Service provides a gRPC API for order routing, execution management, and account state queries. All methods use Protocol Buffers for serialization. - -### Protocol Definition - -The complete protobuf definition is available at `proto/broker_gateway.proto`. - -### Connection - -```rust -use tonic::transport::Channel; -use broker_gateway::broker_gateway_service_client::BrokerGatewayServiceClient; - -// Connect to service -let channel = Channel::from_static("http://localhost:50056") - .connect() - .await?; - -let mut client = BrokerGatewayServiceClient::new(channel); -``` - ---- - -## Authentication - -**MVP**: No authentication (internal service) - -**Phase 2**: mTLS client certificates + API keys - -```rust -// Phase 2: TLS with client cert -use tonic::transport::{Certificate, ClientTlsConfig, Identity}; - -let cert = std::fs::read("client-cert.pem")?; -let key = std::fs::read("client-key.pem")?; -let identity = Identity::from_pem(cert, key); - -let ca_cert = std::fs::read("ca-cert.pem")?; -let ca = Certificate::from_pem(ca_cert); - -let tls = ClientTlsConfig::new() - .identity(identity) - .ca_certificate(ca) - .domain_name("broker-gateway.foxhunt.local"); - -let channel = Channel::from_static("https://broker-gateway:50056") - .tls_config(tls)? - .connect() - .await?; - -let mut client = BrokerGatewayServiceClient::new(channel); -``` - ---- - -## Service Methods - -### RouteOrder - -Submit a new order to the broker. - -#### Request - -```protobuf -message RouteOrderRequest { - string symbol = 1; // ES, NQ, YM, RTY, etc. - OrderSide side = 2; // BUY or SELL - double quantity = 3; // Number of contracts (must be > 0) - OrderType order_type = 4; // MARKET, LIMIT, STOP, STOP_LIMIT - optional double price = 5; // Required for LIMIT orders - optional double stop_price = 6; // Required for STOP orders - string account_id = 7; // AMP account identifier - map metadata = 8; // Optional metadata (strategy, model_name) -} -``` - -#### Response - -```protobuf -message RouteOrderResponse { - string broker_order_id = 1; // Broker's OrderID (filled after ack) - string client_order_id = 2; // Our ClOrdID (UUID) - OrderStatus status = 3; // PENDING_SUBMIT, SUBMITTED, etc. - int64 submitted_at = 4; // Timestamp (nanoseconds) - string message = 5; // Success/error message -} -``` - -#### Example: Market Order - -```rust -use broker_gateway::*; - -let request = RouteOrderRequest { - symbol: "ES".to_string(), - side: OrderSide::Buy as i32, - quantity: 10.0, - order_type: OrderType::Market as i32, - price: None, - stop_price: None, - account_id: "ACCT_001".to_string(), - metadata: [ - ("strategy".to_string(), "momentum".to_string()), - ("model".to_string(), "dqn_v2".to_string()), - ] - .iter() - .cloned() - .collect(), -}; - -let response = client.route_order(request).await?; -let order = response.into_inner(); - -println!("Order submitted:"); -println!(" Client Order ID: {}", order.client_order_id); -println!(" Status: {:?}", OrderStatus::try_from(order.status)?); -println!(" Submitted At: {}", order.submitted_at); - -// Output: -// Order submitted: -// Client Order ID: 550e8400-e29b-41d4-a716-446655440000 -// Status: PENDING_SUBMIT -// Submitted At: 1704812400000000000 -``` - -#### Example: Limit Order - -```rust -let request = RouteOrderRequest { - symbol: "NQ".to_string(), - side: OrderSide::Sell as i32, - quantity: 5.0, - order_type: OrderType::Limit as i32, - price: Some(18500.50), // Required for LIMIT - stop_price: None, - account_id: "ACCT_001".to_string(), - metadata: HashMap::new(), -}; - -let response = client.route_order(request).await?; -``` - -#### Example: Stop-Limit Order - -```rust -let request = RouteOrderRequest { - symbol: "ES".to_string(), - side: OrderSide::Buy as i32, - quantity: 3.0, - order_type: OrderType::StopLimit as i32, - price: Some(5805.00), // Limit price (buy at 5805 after stop triggered) - stop_price: Some(5800.00), // Stop price (trigger at 5800) - account_id: "ACCT_001".to_string(), - metadata: HashMap::new(), -}; - -let response = client.route_order(request).await?; -``` - -#### Validation Rules - -| Field | Validation | -|-------|------------| -| `symbol` | Non-empty, uppercase, valid futures contract | -| `quantity` | Must be > 0 | -| `price` | Required if `order_type = LIMIT` or `STOP_LIMIT` | -| `stop_price` | Required if `order_type = STOP` or `STOP_LIMIT` | -| `account_id` | Non-empty | - -#### Error Codes - -| Code | Description | -|------|-------------| -| `INVALID_ARGUMENT` | Missing required field or invalid value | -| `FAILED_PRECONDITION` | FIX session not active (Phase 2) | -| `RESOURCE_EXHAUSTED` | Rate limit exceeded | -| `INTERNAL` | Database error or unexpected failure | - ---- - -### CancelOrder - -Cancel an existing order. - -#### Request - -```protobuf -message CancelOrderRequest { - string client_order_id = 1; // Order to cancel (required) - string account_id = 2; // Account verification (required) -} -``` - -#### Response - -```protobuf -message CancelOrderResponse { - bool success = 1; // True if cancel request accepted - string message = 2; // Confirmation message or error - OrderStatus new_status = 3; // CANCEL_PENDING or CANCELLED -} -``` - -#### Example - -```rust -let request = CancelOrderRequest { - client_order_id: "550e8400-e29b-41d4-a716-446655440000".to_string(), - account_id: "ACCT_001".to_string(), -}; - -let response = client.cancel_order(request).await?; -let cancel = response.into_inner(); - -if cancel.success { - println!("Cancel request accepted: {}", cancel.message); - println!("New status: {:?}", OrderStatus::try_from(cancel.new_status)?); -} else { - println!("Cancel request failed: {}", cancel.message); -} - -// Output: -// Cancel request accepted: Cancel request queued (MVP: no FIX send). Order: 550e8400-... -// New status: CANCEL_PENDING -``` - -#### Cancellable States - -Only orders in these states can be cancelled: -- `PENDING_SUBMIT` -- `SUBMITTED` -- `PARTIALLY_FILLED` - -#### Error Codes - -| Code | Description | -|------|-------------| -| `NOT_FOUND` | Order not found or account_id mismatch | -| `FAILED_PRECONDITION` | Order already filled, cancelled, or rejected | -| `INTERNAL` | Database error | - ---- - -### GetAccountState - -Retrieve current account balance and margin information. - -#### Request - -```protobuf -message GetAccountStateRequest { - string account_id = 1; // AMP account identifier -} -``` - -#### Response - -```protobuf -message GetAccountStateResponse { - string account_id = 1; - double cash_balance = 2; // Cash balance (USD) - double equity = 3; // Cash + unrealized P&L - double margin_used = 4; // Margin locked by open positions - double margin_available = 5; // Available margin for new positions - double buying_power = 6; // Margin available * leverage - double unrealized_pnl = 7; // Unrealized profit/loss - double realized_pnl = 8; // Realized profit/loss (today) - int64 last_updated = 9; // Timestamp (nanoseconds) -} -``` - -#### Example - -```rust -let request = GetAccountStateRequest { - account_id: "ACCT_001".to_string(), -}; - -let response = client.get_account_state(request).await?; -let state = response.into_inner(); - -println!("Account: {}", state.account_id); -println!("Cash Balance: ${:.2}", state.cash_balance); -println!("Equity: ${:.2}", state.equity); -println!("Margin Used: ${:.2}", state.margin_used); -println!("Margin Available: ${:.2}", state.margin_available); -println!("Buying Power: ${:.2}", state.buying_power); -println!("Unrealized P&L: ${:.2}", state.unrealized_pnl); -println!("Realized P&L: ${:.2}", state.realized_pnl); - -// Output: -// Account: ACCT_001 -// Cash Balance: $100000.00 -// Equity: $100000.00 -// Margin Used: $0.00 -// Margin Available: $100000.00 -// Buying Power: $400000.00 -// Unrealized P&L: $0.00 -// Realized P&L: $0.00 -``` - -#### MVP Behavior - -Returns placeholder data (cash_balance = $100,000, 4x leverage). - -**Phase 2**: Queries CQG broker via FIX `CollateralInquiry` (MsgType=BB). - -#### Error Codes - -| Code | Description | -|------|-------------| -| `NOT_FOUND` | Account not found | -| `UNAVAILABLE` | Broker connection unavailable (Phase 2) | -| `INTERNAL` | Database error | - ---- - -### GetPositions - -Retrieve current open positions. - -#### Request - -```protobuf -message GetPositionsRequest { - string account_id = 1; // AMP account identifier - optional string symbol = 2; // Filter by symbol (optional) -} -``` - -#### Response - -```protobuf -message GetPositionsResponse { - repeated Position positions = 1; // List of positions - double total_equity = 2; // Total account equity - double total_exposure = 3; // Sum of abs(position_value) - double leverage_ratio = 4; // total_exposure / total_equity - int64 timestamp = 5; // Timestamp (nanoseconds) -} - -message Position { - string symbol = 1; // ES, NQ, etc. - double quantity = 2; // Positive = long, negative = short - double average_price = 3; // Average entry price - double market_value = 4; // quantity * current_price - double unrealized_pnl = 5; // (current_price - avg_price) * quantity -} -``` - -#### Example: All Positions - -```rust -let request = GetPositionsRequest { - account_id: "ACCT_001".to_string(), - symbol: None, // All symbols -}; - -let response = client.get_positions(request).await?; -let positions_response = response.into_inner(); - -println!("Total Equity: ${:.2}", positions_response.total_equity); -println!("Total Exposure: ${:.2}", positions_response.total_exposure); -println!("Leverage: {:.2}x", positions_response.leverage_ratio); -println!("\nPositions:"); - -for pos in &positions_response.positions { - println!(" {} x {} @ ${:.2} | Market: ${:.2} | P&L: ${:.2}", - pos.symbol, - pos.quantity, - pos.average_price, - pos.market_value, - pos.unrealized_pnl - ); -} - -// Output: -// Total Equity: $102350.00 -// Total Exposure: $290000.00 -// Leverage: 2.83x -// -// Positions: -// ES x 10 @ $5800.00 | Market: $58050.00 | P&L: $250.00 -// NQ x -5 @ $18500.00 | Market: $-92000.00 | P&L: $-100.00 -``` - -#### Example: Single Symbol - -```rust -let request = GetPositionsRequest { - account_id: "ACCT_001".to_string(), - symbol: Some("ES".to_string()), -}; - -let response = client.get_positions(request).await?; -// Returns only ES positions -``` - -#### MVP Behavior - -Returns empty positions list. - -**Phase 2**: Queries CQG broker via FIX `RequestForPositions` (MsgType=AN). - -#### Error Codes - -| Code | Description | -|------|-------------| -| `NOT_FOUND` | Account not found | -| `UNAVAILABLE` | Broker connection unavailable (Phase 2) | -| `INTERNAL` | Database error | - ---- - -### GetSessionStatus - -Retrieve FIX session status and health metrics. - -#### Request - -```protobuf -message GetSessionStatusRequest { - optional string session_id = 1; // Optional: default to active session -} -``` - -#### Response - -```protobuf -message GetSessionStatusResponse { - string session_id = 1; // Session identifier (e.g., "FOXHUNT-CQG") - SessionState state = 2; // DISCONNECTED, CONNECTED, ACTIVE, etc. - int64 sender_seq_num = 3; // Current outgoing MsgSeqNum - int64 target_seq_num = 4; // Expected incoming MsgSeqNum - int64 last_heartbeat_sent = 5; // Timestamp (nanoseconds) - int64 last_heartbeat_received = 6; // Timestamp (nanoseconds) - double heartbeat_rtt_ms = 7; // Round-trip time (milliseconds) - int64 connected_at = 8; // Connection timestamp (nanoseconds) - map details = 9; // Additional info -} - -enum SessionState { - SESSION_STATE_DISCONNECTED = 0; - SESSION_STATE_CONNECTED = 1; - SESSION_STATE_LOGGING_IN = 2; - SESSION_STATE_ACTIVE = 3; - SESSION_STATE_LOGGING_OUT = 4; -} -``` - -#### Example - -```rust -let request = GetSessionStatusRequest { - session_id: None, // Use default session -}; - -let response = client.get_session_status(request).await?; -let status = response.into_inner(); - -println!("Session: {}", status.session_id); -println!("State: {:?}", SessionState::try_from(status.state)?); -println!("Sender Seq: {}", status.sender_seq_num); -println!("Target Seq: {}", status.target_seq_num); -println!("Heartbeat RTT: {:.2} ms", status.heartbeat_rtt_ms); - -// Output: -// Session: FOXHUNT-CQG -// State: ACTIVE -// Sender Seq: 5432 -// Target Seq: 5398 -// Heartbeat RTT: 23.45 ms -``` - -#### MVP Behavior - -Returns simulated session state (ACTIVE, sequence = 1). - -**Phase 2**: Returns actual FIX session metrics. - -#### Error Codes - -| Code | Description | -|------|-------------| -| `NOT_FOUND` | Session ID not found | -| `INTERNAL` | Database error | - ---- - -### StreamExecutions - -Stream real-time execution reports from the broker. - -#### Request - -```protobuf -message StreamExecutionsRequest { - optional string account_id = 1; // Filter by account (optional) - optional string symbol = 2; // Filter by symbol (optional) -} -``` - -#### Response (Stream) - -```protobuf -message ExecutionEvent { - string execution_id = 1; // ExecID (Tag 17) - string broker_order_id = 2; // OrderID (Tag 37) - string client_order_id = 3; // ClOrdID (Tag 11) - string symbol = 4; - OrderSide side = 5; - ExecutionType exec_type = 6; // NEW, TRADE, CANCELED, REJECTED - OrderStatus order_status = 7; // Order status after this execution - double last_qty = 8; // Quantity filled (Tag 32) - double last_price = 9; // Fill price (Tag 31) - double cum_qty = 10; // Total filled (Tag 14) - double avg_price = 11; // Average fill price (Tag 6) - int64 transact_time = 12; // Execution timestamp - optional string text = 13; // Reject reason (if applicable) -} - -enum ExecutionType { - EXECUTION_TYPE_UNSPECIFIED = 0; - EXECUTION_TYPE_NEW = 1; // Order accepted by broker - EXECUTION_TYPE_TRADE = 2; // Partial or full fill - EXECUTION_TYPE_CANCELED = 3; // Order canceled - EXECUTION_TYPE_REJECTED = 4; // Order rejected -} -``` - -#### Example - -```rust -use tokio_stream::StreamExt; - -let request = StreamExecutionsRequest { - account_id: Some("ACCT_001".to_string()), - symbol: None, // All symbols -}; - -let mut stream = client.stream_executions(request).await?.into_inner(); - -println!("Streaming executions..."); - -while let Some(event) = stream.next().await { - match event { - Ok(exec) => { - println!("\n--- Execution Event ---"); - println!("Execution ID: {}", exec.execution_id); - println!("Order ID: {} (Broker: {})", exec.client_order_id, exec.broker_order_id); - println!("Symbol: {} {} {}", exec.symbol, - if exec.side == OrderSide::Buy as i32 { "BUY" } else { "SELL" }, - exec.last_qty); - println!("Exec Type: {:?}", ExecutionType::try_from(exec.exec_type)?); - println!("Order Status: {:?}", OrderStatus::try_from(exec.order_status)?); - - if exec.exec_type == ExecutionType::Trade as i32 { - println!("Fill: {} @ ${:.2}", exec.last_qty, exec.last_price); - println!("Cumulative: {} @ ${:.2} avg", exec.cum_qty, exec.avg_price); - } - - if let Some(text) = exec.text { - println!("Message: {}", text); - } - } - Err(e) => { - eprintln!("Stream error: {}", e); - break; - } - } -} - -// Output: -// Streaming executions... -// -// --- Execution Event --- -// Execution ID: EXEC_789 -// Order ID: 550e8400-... (Broker: BROKER_123) -// Symbol: ES BUY 10 -// Exec Type: TRADE -// Order Status: FILLED -// Fill: 10 @ $5800.25 -// Cumulative: 10 @ $5800.25 avg -``` - -#### MVP Behavior - -Stream closes immediately (no executions). - -**Phase 2**: Streams real-time FIX `ExecutionReport` messages. - -#### Error Codes - -| Code | Description | -|------|-------------| -| `UNAVAILABLE` | FIX session not active (Phase 2) | -| `INTERNAL` | Stream initialization failed | - ---- - -### HealthCheck - -Check service health (database, FIX session). - -#### Request - -```protobuf -message HealthCheckRequest {} -``` - -#### Response - -```protobuf -message HealthCheckResponse { - bool healthy = 1; // True if service is healthy - string message = 2; // Status message - map details = 3; // Component health details -} -``` - -#### Example - -```rust -let request = HealthCheckRequest {}; -let response = client.health_check(request).await?; -let health = response.into_inner(); - -println!("Service Healthy: {}", health.healthy); -println!("Message: {}", health.message); -println!("Details:"); -for (key, value) in &health.details { - println!(" {}: {}", key, value); -} - -// Output: -// Service Healthy: true -// Message: Broker Gateway Service is healthy (MVP mode) -// Details: -// database: true -// mvp_mode: true -// fix_session: not_implemented -``` - -#### Error Codes - -Always returns `OK`. Check `healthy` field in response. - ---- - -## Data Types - -### Enums - -#### OrderSide - -```protobuf -enum OrderSide { - ORDER_SIDE_UNSPECIFIED = 0; - ORDER_SIDE_BUY = 1; - ORDER_SIDE_SELL = 2; -} -``` - -#### OrderType - -```protobuf -enum OrderType { - ORDER_TYPE_UNSPECIFIED = 0; - ORDER_TYPE_MARKET = 1; - ORDER_TYPE_LIMIT = 2; - ORDER_TYPE_STOP = 3; - ORDER_TYPE_STOP_LIMIT = 4; -} -``` - -#### OrderStatus - -```protobuf -enum OrderStatus { - ORDER_STATUS_UNSPECIFIED = 0; - ORDER_STATUS_PENDING_SUBMIT = 1; // Order created, not sent - ORDER_STATUS_SUBMITTED = 2; // Sent to broker - ORDER_STATUS_PARTIALLY_FILLED = 3; // Partially filled - ORDER_STATUS_FILLED = 4; // Fully filled - ORDER_STATUS_CANCEL_PENDING = 5; // Cancel request sent - ORDER_STATUS_CANCELLED = 6; // Cancelled by broker - ORDER_STATUS_REJECTED = 7; // Rejected by broker -} -``` - -#### ExecutionType - -```protobuf -enum ExecutionType { - EXECUTION_TYPE_UNSPECIFIED = 0; - EXECUTION_TYPE_NEW = 1; // Order accepted - EXECUTION_TYPE_TRADE = 2; // Fill (partial or full) - EXECUTION_TYPE_CANCELED = 3; // Order canceled - EXECUTION_TYPE_REJECTED = 4; // Order rejected -} -``` - -#### SessionState - -```protobuf -enum SessionState { - SESSION_STATE_DISCONNECTED = 0; - SESSION_STATE_CONNECTED = 1; - SESSION_STATE_LOGGING_IN = 2; - SESSION_STATE_ACTIVE = 3; - SESSION_STATE_LOGGING_OUT = 4; -} -``` - ---- - -## Error Codes - -### gRPC Status Codes - -| Code | HTTP | Description | Retry | -|------|------|-------------|-------| -| `OK` | 200 | Success | - | -| `INVALID_ARGUMENT` | 400 | Invalid request parameters | No | -| `NOT_FOUND` | 404 | Resource not found | No | -| `ALREADY_EXISTS` | 409 | Duplicate order ID | No | -| `FAILED_PRECONDITION` | 400 | Order not cancellable | No | -| `RESOURCE_EXHAUSTED` | 429 | Rate limit exceeded | Yes (backoff) | -| `UNAVAILABLE` | 503 | Service unavailable | Yes (backoff) | -| `INTERNAL` | 500 | Internal server error | Yes (limited) | -| `DEADLINE_EXCEEDED` | 504 | Request timeout | Yes (once) | - -### Error Details - -Errors include structured details in metadata: - -```rust -use tonic::{Code, Status}; - -// Example error response -let status = Status::new( - Code::InvalidArgument, - "Price is required for LIMIT orders" -); - -// Client error handling -match client.route_order(request).await { - Ok(response) => { /* ... */ }, - Err(e) => { - match e.code() { - Code::InvalidArgument => { - eprintln!("Validation error: {}", e.message()); - // Don't retry - } - Code::Unavailable => { - eprintln!("Service unavailable: {}", e.message()); - // Retry with backoff - } - _ => { - eprintln!("Unexpected error: {} ({})", e.message(), e.code()); - } - } - } -} -``` - ---- - -## Rate Limits - -### MVP - -No rate limits. - -### Phase 2 - -| Method | Limit | Window | -|--------|-------|--------| -| RouteOrder | 100 req/sec | Per account | -| CancelOrder | 50 req/sec | Per account | -| GetAccountState | 10 req/sec | Per account | -| GetPositions | 10 req/sec | Per account | -| GetSessionStatus | 5 req/sec | Global | -| StreamExecutions | 1 connection | Per account | - -Rate limit exceeded returns `RESOURCE_EXHAUSTED` (HTTP 429). - ---- - -## Client Examples - -### Complete Order Lifecycle - -```rust -use broker_gateway::broker_gateway_service_client::BrokerGatewayServiceClient; -use broker_gateway::*; -use tonic::transport::Channel; -use tokio_stream::StreamExt; - -#[tokio::main] -async fn main() -> Result<(), Box> { - // 1. Connect to service - let channel = Channel::from_static("http://localhost:50056") - .connect() - .await?; - let mut client = BrokerGatewayServiceClient::new(channel); - - // 2. Stream executions in background - let mut stream_client = client.clone(); - tokio::spawn(async move { - let request = StreamExecutionsRequest { - account_id: Some("ACCT_001".to_string()), - symbol: None, - }; - let mut stream = stream_client.stream_executions(request).await.unwrap().into_inner(); - while let Some(event) = stream.next().await { - if let Ok(exec) = event { - println!("Execution: {} {} @ ${}", exec.symbol, exec.last_qty, exec.last_price); - } - } - }); - - // 3. Submit market order - let request = RouteOrderRequest { - symbol: "ES".to_string(), - side: OrderSide::Buy as i32, - quantity: 10.0, - order_type: OrderType::Market as i32, - price: None, - stop_price: None, - account_id: "ACCT_001".to_string(), - metadata: HashMap::new(), - }; - - let response = client.route_order(request).await?; - let order = response.into_inner(); - println!("Order submitted: {}", order.client_order_id); - - // 4. Wait 5 seconds (simulating fill delay) - tokio::time::sleep(tokio::time::Duration::from_secs(5)).await; - - // 5. Get updated positions - let request = GetPositionsRequest { - account_id: "ACCT_001".to_string(), - symbol: Some("ES".to_string()), - }; - let response = client.get_positions(request).await?; - let positions = response.into_inner(); - for pos in positions.positions { - println!("Position: {} x {}", pos.symbol, pos.quantity); - } - - Ok(()) -} -``` - -### Error Handling with Retries - -```rust -use std::time::Duration; -use tokio::time::sleep; - -async fn route_order_with_retry( - client: &mut BrokerGatewayServiceClient, - request: RouteOrderRequest, - max_retries: u32, -) -> Result { - let mut attempt = 0; - let mut delay = Duration::from_millis(100); - - loop { - match client.route_order(request.clone()).await { - Ok(response) => return Ok(response.into_inner()), - Err(e) => { - let should_retry = matches!( - e.code(), - Code::Unavailable | Code::DeadlineExceeded | Code::ResourceExhausted - ); - - if should_retry && attempt < max_retries { - println!("Retry attempt {}/{} after {:?}", attempt + 1, max_retries, delay); - sleep(delay).await; - attempt += 1; - delay = Duration::min(delay * 2, Duration::from_secs(10)); - } else { - return Err(e); - } - } - } - } -} -``` - -### Bulk Order Submission - -```rust -async fn submit_bulk_orders( - client: &mut BrokerGatewayServiceClient, - orders: Vec, -) -> Vec> { - let mut results = Vec::new(); - - for order in orders { - let result = match client.route_order(order.clone()).await { - Ok(response) => Ok(response.into_inner().client_order_id), - Err(e) => Err(e.message().to_string()), - }; - results.push(result); - } - - results -} - -// Example usage -let orders = vec![ - RouteOrderRequest { symbol: "ES".to_string(), /* ... */ }, - RouteOrderRequest { symbol: "NQ".to_string(), /* ... */ }, - RouteOrderRequest { symbol: "YM".to_string(), /* ... */ }, -]; - -let results = submit_bulk_orders(&mut client, orders).await; -for (i, result) in results.iter().enumerate() { - match result { - Ok(order_id) => println!("Order {} submitted: {}", i, order_id), - Err(e) => println!("Order {} failed: {}", i, e), - } -} -``` - ---- - -## Changelog - -### v0.1.0 (2025-01-09) - -**MVP Release**: -- gRPC API endpoints (7 methods) -- Database persistence -- Request validation -- Health checks - -**Limitations**: -- No FIX protocol (placeholder responses) -- No real broker connectivity -- StreamExecutions closes immediately - -### v0.2.0 (Planned - Phase 2) - -**Full FIX Integration**: -- FIX 4.2/4.4 protocol implementation -- CQG broker connectivity -- Real-time execution streaming -- Position reconciliation -- Session management (LOGON, LOGOUT, Heartbeat) -- Sequence number tracking - ---- - -## Support - -For issues or questions: -- **Internal**: Slack #broker-gateway-support -- **Email**: ops@foxhunt.trading -- **Runbook**: See [TROUBLESHOOTING.md](TROUBLESHOOTING.md) diff --git a/services/broker_gateway_service/docs/DEPLOYMENT.md b/services/broker_gateway_service/docs/DEPLOYMENT.md deleted file mode 100644 index acf474588..000000000 --- a/services/broker_gateway_service/docs/DEPLOYMENT.md +++ /dev/null @@ -1,990 +0,0 @@ -# Broker Gateway Service Deployment Guide - -**Version**: 0.1.0 -**Last Updated**: 2025-01-09 - -## Table of Contents - -- [Prerequisites](#prerequisites) -- [Docker Deployment](#docker-deployment) -- [Kubernetes Deployment](#kubernetes-deployment) -- [Monitoring Setup](#monitoring-setup) -- [Production Readiness Checklist](#production-readiness-checklist) -- [Rollback Procedure](#rollback-procedure) -- [Scaling Guidelines](#scaling-guidelines) - ---- - -## Prerequisites - -### Infrastructure Requirements - -| Component | Minimum | Recommended | Purpose | -|-----------|---------|-------------|---------| -| **CPU** | 2 cores | 4 cores | gRPC server, FIX session | -| **Memory** | 512 MB | 2 GB | Message buffers, connection pools | -| **Disk** | 10 GB | 50 GB | Logs, metrics, temp storage | -| **Network** | 100 Mbps | 1 Gbps | FIX messages, gRPC traffic | - -### External Dependencies - -```bash -# 1. PostgreSQL 14+ -psql --version -# postgresql (PostgreSQL) 14.10 - -# 2. Redis 7+ -redis-cli --version -# redis-cli 7.2.3 - -# 3. Docker 24+ (for containerized deployment) -docker --version -# Docker version 24.0.7, build afdd53b - -# 4. Kubernetes 1.28+ (for K8s deployment) -kubectl version --short -# Client Version: v1.28.4 -# Server Version: v1.28.4 -``` - -### CQG Credentials (Phase 2) - -1. **FIX Account Setup**: - - Contact AMP Futures: https://www.ampfutures.com/ - - Request FIX API access - - Obtain `SenderCompID`, `TargetCompID`, username, password - -2. **Firewall Rules**: - - Whitelist your IP with CQG - - Open outbound TCP to CQG FIX endpoint (port provided by CQG) - -3. **Test Credentials** (UAT environment): - ```bash - # Verify FIX connectivity (telnet) - telnet fix-uat.cqg.com - # Should see: Connected to fix-uat.cqg.com - ``` - -### Database Setup - -```bash -# 1. Create database -psql -U postgres -c "CREATE DATABASE foxhunt;" -psql -U postgres -c "CREATE USER foxhunt WITH PASSWORD 'foxhunt_dev_password';" -psql -U postgres -c "GRANT ALL PRIVILEGES ON DATABASE foxhunt TO foxhunt;" - -# 2. Run migrations -cd /opt/foxhunt -export DATABASE_URL=postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -cargo sqlx migrate run - -# 3. Verify tables -psql -U foxhunt -c "\dt" -# Expected: -# Schema | Name | Type | Owner -# --------+---------------------+-------+--------- -# public | broker_orders | table | foxhunt -# public | broker_executions | table | foxhunt -# public | fix_sessions | table | foxhunt -``` - ---- - -## Docker Deployment - -### Single Container (Development) - -#### 1. Build Docker Image - -```dockerfile -# Dockerfile (multi-stage build) -FROM rust:1.75-slim as builder - -WORKDIR /build - -# Install build dependencies -RUN apt-get update && apt-get install -y \ - pkg-config \ - libssl-dev \ - protobuf-compiler \ - && rm -rf /var/lib/apt/lists/* - -# Copy workspace files -COPY Cargo.toml Cargo.lock ./ -COPY services/broker_gateway_service ./services/broker_gateway_service -COPY common ./common -COPY config ./config - -# Build release binary -RUN cargo build --release -p broker_gateway_service - -# ============================================================================ -# Runtime image -# ============================================================================ -FROM debian:bookworm-slim - -RUN apt-get update && apt-get install -y \ - ca-certificates \ - libssl3 \ - curl \ - && rm -rf /var/lib/apt/lists/* - -# Copy binary from builder -COPY --from=builder /build/target/release/broker_gateway_service /usr/local/bin/ - -# Create non-root user -RUN useradd -m -u 1000 foxhunt && \ - mkdir -p /var/log/broker_gateway && \ - chown -R foxhunt:foxhunt /var/log/broker_gateway - -USER foxhunt - -EXPOSE 50056 8086 9096 - -HEALTHCHECK --interval=30s --timeout=5s --retries=3 \ - CMD curl -f http://localhost:8086/health || exit 1 - -ENTRYPOINT ["/usr/local/bin/broker_gateway_service"] -``` - -#### 2. Build and Run - -```bash -# Build image -docker build -t foxhunt/broker-gateway:v0.1.0 \ - -f services/broker_gateway_service/Dockerfile . - -# Run container -docker run -d \ - --name broker-gateway \ - --network foxhunt-network \ - -p 50056:50056 \ - -p 8086:8086 \ - -p 9096:9096 \ - -e DATABASE_URL=postgresql://foxhunt:foxhunt_dev_password@postgres:5432/foxhunt \ - -e REDIS_URL=redis://redis:6379 \ - -e RUST_LOG=info,broker_gateway_service=debug \ - -e LOG_FORMAT=json \ - -v /var/log/broker_gateway:/var/log/broker_gateway \ - foxhunt/broker-gateway:v0.1.0 - -# Verify health -curl http://localhost:8086/health -``` - ---- - -### Docker Compose (Full Stack) - -```yaml -# docker-compose.yml -version: '3.8' - -services: - postgres: - image: postgres:14-alpine - environment: - POSTGRES_USER: foxhunt - POSTGRES_PASSWORD: foxhunt_dev_password - POSTGRES_DB: foxhunt - ports: - - "5432:5432" - volumes: - - postgres_data:/var/lib/postgresql/data - healthcheck: - test: ["CMD-SHELL", "pg_isready -U foxhunt"] - interval: 10s - timeout: 5s - retries: 5 - - redis: - image: redis:7-alpine - ports: - - "6379:6379" - volumes: - - redis_data:/data - healthcheck: - test: ["CMD", "redis-cli", "ping"] - interval: 10s - timeout: 5s - retries: 5 - - broker-gateway: - build: - context: . - dockerfile: services/broker_gateway_service/Dockerfile - depends_on: - postgres: - condition: service_healthy - redis: - condition: service_healthy - ports: - - "50056:50056" - - "8086:8086" - - "9096:9096" - environment: - DATABASE_URL: postgresql://foxhunt:foxhunt_dev_password@postgres:5432/foxhunt - REDIS_URL: redis://redis:6379 - GRPC_PORT: 50056 - HEALTH_PORT: 8086 - METRICS_PORT: 9096 - RUST_LOG: info,broker_gateway_service=debug - LOG_FORMAT: json - # FIX credentials (Phase 2) - FIX_SENDER_COMP_ID: ${FIX_SENDER_COMP_ID} - FIX_TARGET_COMP_ID: ${FIX_TARGET_COMP_ID} - FIX_USERNAME: ${FIX_USERNAME} - FIX_PASSWORD: ${FIX_PASSWORD} - FIX_HOST: ${FIX_HOST} - FIX_PORT: ${FIX_PORT} - volumes: - - logs:/var/log/broker_gateway - healthcheck: - test: ["CMD", "curl", "-f", "http://localhost:8086/health"] - interval: 30s - timeout: 5s - retries: 3 - restart: unless-stopped - - prometheus: - image: prom/prometheus:latest - ports: - - "9090:9090" - volumes: - - ./monitoring/prometheus.yml:/etc/prometheus/prometheus.yml - - prometheus_data:/prometheus - command: - - '--config.file=/etc/prometheus/prometheus.yml' - - '--storage.tsdb.path=/prometheus' - restart: unless-stopped - - grafana: - image: grafana/grafana:latest - ports: - - "3000:3000" - environment: - GF_SECURITY_ADMIN_PASSWORD: admin - GF_INSTALL_PLUGINS: grafana-piechart-panel - volumes: - - ./monitoring/grafana/dashboards:/var/lib/grafana/dashboards - - grafana_data:/var/lib/grafana - restart: unless-stopped - -volumes: - postgres_data: - redis_data: - logs: - prometheus_data: - grafana_data: -``` - -```bash -# Deploy full stack -docker-compose up -d - -# View logs -docker-compose logs -f broker-gateway - -# Stop services -docker-compose down -``` - ---- - -## Kubernetes Deployment - -### Namespace and ConfigMap - -```yaml -# 1-namespace.yaml -apiVersion: v1 -kind: Namespace -metadata: - name: foxhunt - labels: - name: foxhunt - env: production -``` - -```yaml -# 2-configmap.yaml -apiVersion: v1 -kind: ConfigMap -metadata: - name: broker-gateway-config - namespace: foxhunt -data: - GRPC_PORT: "50056" - HEALTH_PORT: "8086" - METRICS_PORT: "9096" - RUST_LOG: "info,broker_gateway_service=debug" - LOG_FORMAT: "json" - DATABASE_URL: "postgresql://foxhunt:PASSWORD@postgres.foxhunt.svc.cluster.local:5432/foxhunt" - REDIS_URL: "redis://redis.foxhunt.svc.cluster.local:6379" -``` - -### Secret (FIX Credentials) - -```yaml -# 3-secret.yaml -apiVersion: v1 -kind: Secret -metadata: - name: broker-gateway-secrets - namespace: foxhunt -type: Opaque -stringData: - FIX_SENDER_COMP_ID: "FOXHUNT_CLIENT" - FIX_TARGET_COMP_ID: "CQG" - FIX_USERNAME: "your_username" - FIX_PASSWORD: "your_password" - FIX_HOST: "fix.cqg.com" - FIX_PORT: "xxxx" -``` - -```bash -# Create secret from file -kubectl create secret generic broker-gateway-secrets \ - --from-env-file=.env.production \ - --namespace=foxhunt -``` - -### StatefulSet - -```yaml -# 4-statefulset.yaml -apiVersion: apps/v1 -kind: StatefulSet -metadata: - name: broker-gateway - namespace: foxhunt - labels: - app: broker-gateway -spec: - serviceName: broker-gateway - replicas: 2 - selector: - matchLabels: - app: broker-gateway - template: - metadata: - labels: - app: broker-gateway - version: v0.1.0 - annotations: - prometheus.io/scrape: "true" - prometheus.io/port: "9096" - prometheus.io/path: "/metrics" - spec: - affinity: - podAntiAffinity: - preferredDuringSchedulingIgnoredDuringExecution: - - weight: 100 - podAffinityTerm: - labelSelector: - matchLabels: - app: broker-gateway - topologyKey: kubernetes.io/hostname - containers: - - name: broker-gateway - image: foxhunt/broker-gateway:v0.1.0 - imagePullPolicy: Always - ports: - - name: grpc - containerPort: 50056 - protocol: TCP - - name: health - containerPort: 8086 - protocol: TCP - - name: metrics - containerPort: 9096 - protocol: TCP - envFrom: - - configMapRef: - name: broker-gateway-config - - secretRef: - name: broker-gateway-secrets - resources: - requests: - cpu: 500m - memory: 512Mi - limits: - cpu: 2000m - memory: 2Gi - livenessProbe: - httpGet: - path: /health - port: health - initialDelaySeconds: 30 - periodSeconds: 30 - timeoutSeconds: 5 - failureThreshold: 3 - readinessProbe: - httpGet: - path: /health - port: health - initialDelaySeconds: 10 - periodSeconds: 10 - timeoutSeconds: 5 - failureThreshold: 3 - volumeMounts: - - name: logs - mountPath: /var/log/broker_gateway - volumeClaimTemplates: - - metadata: - name: logs - spec: - accessModes: ["ReadWriteOnce"] - storageClassName: fast-ssd - resources: - requests: - storage: 50Gi -``` - -### Service - -```yaml -# 5-service.yaml -apiVersion: v1 -kind: Service -metadata: - name: broker-gateway - namespace: foxhunt - labels: - app: broker-gateway -spec: - type: ClusterIP - selector: - app: broker-gateway - ports: - - name: grpc - port: 50056 - targetPort: grpc - protocol: TCP - - name: health - port: 8086 - targetPort: health - protocol: TCP - - name: metrics - port: 9096 - targetPort: metrics - protocol: TCP - sessionAffinity: ClientIP - sessionAffinityConfig: - clientIP: - timeoutSeconds: 3600 -``` - -### Horizontal Pod Autoscaler - -```yaml -# 6-hpa.yaml -apiVersion: autoscaling/v2 -kind: HorizontalPodAutoscaler -metadata: - name: broker-gateway-hpa - namespace: foxhunt -spec: - scaleTargetRef: - apiVersion: apps/v1 - kind: StatefulSet - name: broker-gateway - minReplicas: 2 - maxReplicas: 10 - metrics: - - type: Resource - resource: - name: cpu - target: - type: Utilization - averageUtilization: 70 - - type: Resource - resource: - name: memory - target: - type: Utilization - averageUtilization: 80 - - type: Pods - pods: - metric: - name: broker_order_submit_latency_seconds_p99 - target: - type: Value - averageValue: "0.005" # 5ms P99 latency -``` - -### Deploy to Kubernetes - -```bash -# 1. Create namespace -kubectl apply -f 1-namespace.yaml - -# 2. Create ConfigMap and Secret -kubectl apply -f 2-configmap.yaml -kubectl apply -f 3-secret.yaml - -# 3. Deploy StatefulSet and Service -kubectl apply -f 4-statefulset.yaml -kubectl apply -f 5-service.yaml - -# 4. Deploy HPA -kubectl apply -f 6-hpa.yaml - -# 5. Verify deployment -kubectl get all -n foxhunt - -# Expected output: -# NAME READY STATUS RESTARTS AGE -# pod/broker-gateway-0 1/1 Running 0 2m -# pod/broker-gateway-1 1/1 Running 0 1m -# -# NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) -# service/broker-gateway ClusterIP 10.96.123.45 50056/TCP,8086/TCP,9096/TCP -# -# NAME READY AGE -# statefulset.apps/broker-gateway 2/2 2m - -# 6. Check logs -kubectl logs -f broker-gateway-0 -n foxhunt - -# 7. Port-forward for testing -kubectl port-forward service/broker-gateway 50056:50056 -n foxhunt -``` - ---- - -## Monitoring Setup - -### Prometheus Configuration - -```yaml -# monitoring/prometheus.yml -global: - scrape_interval: 15s - evaluation_interval: 15s - -scrape_configs: - - job_name: 'broker-gateway' - kubernetes_sd_configs: - - role: pod - namespaces: - names: - - foxhunt - relabel_configs: - - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape] - action: keep - regex: true - - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port] - action: replace - target_label: __address__ - regex: ([^:]+)(?::\d+)?;(\d+) - replacement: $1:$2 - - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path] - action: replace - target_label: __metrics_path__ - regex: (.+) - -rule_files: - - '/etc/prometheus/rules/*.yml' -``` - -### Alert Rules - -```yaml -# monitoring/prometheus/rules/broker-gateway.yml -groups: - - name: broker-gateway - interval: 30s - rules: - # Alert: FIX session down - - alert: FixSessionDown - expr: fix_session_state{session_id="FOXHUNT-CQG"} != 3 - for: 1m - labels: - severity: critical - service: broker-gateway - annotations: - summary: "FIX session not active" - description: "FIX session {{ $labels.session_id }} is in state {{ $value }} (expected: 3=ACTIVE)" - - # Alert: High order rejection rate - - alert: HighOrderRejectionRate - expr: | - rate(broker_orders_total{status="REJECTED"}[5m]) - / rate(broker_orders_total[5m]) > 0.1 - for: 5m - labels: - severity: warning - service: broker-gateway - annotations: - summary: "Order rejection rate > 10%" - description: "{{ $value | humanizePercentage }} of orders are being rejected" - - # Alert: Position discrepancy - - alert: PositionDiscrepancy - expr: position_reconciliation_discrepancies > 0 - for: 1m - labels: - severity: warning - service: broker-gateway - annotations: - summary: "Position mismatch detected" - description: "{{ $value }} position discrepancies between broker and database" - - # Alert: High latency - - alert: HighOrderSubmitLatency - expr: | - histogram_quantile(0.99, - rate(broker_order_submit_latency_seconds_bucket[5m]) - ) > 0.010 - for: 5m - labels: - severity: warning - service: broker-gateway - annotations: - summary: "Order submission latency P99 > 10ms" - description: "P99 latency is {{ $value }}s (target: <5ms)" - - # Alert: Database connection pool exhausted - - alert: DatabasePoolExhausted - expr: db_pool_idle_connections == 0 and db_pool_wait_time_ms > 50 - for: 2m - labels: - severity: warning - service: broker-gateway - annotations: - summary: "Database connection pool exhausted" - description: "No idle connections, wait time {{ $value }}ms" -``` - -### Grafana Dashboard - -```json -{ - "dashboard": { - "title": "Broker Gateway Service", - "panels": [ - { - "title": "Order Flow", - "targets": [ - { - "expr": "rate(broker_orders_total[1m])", - "legendFormat": "Orders/sec - {{status}}" - } - ] - }, - { - "title": "FIX Session State", - "targets": [ - { - "expr": "fix_session_state", - "legendFormat": "{{session_id}}" - } - ], - "gauge": { - "minValue": 0, - "maxValue": 4, - "thresholds": [ - { "value": 0, "color": "red" }, - { "value": 3, "color": "green" } - ] - } - }, - { - "title": "Order Submission Latency", - "targets": [ - { - "expr": "histogram_quantile(0.50, rate(broker_order_submit_latency_seconds_bucket[5m]))", - "legendFormat": "P50" - }, - { - "expr": "histogram_quantile(0.99, rate(broker_order_submit_latency_seconds_bucket[5m]))", - "legendFormat": "P99" - } - ] - }, - { - "title": "Error Rate", - "targets": [ - { - "expr": "rate(broker_orders_total{status=\"REJECTED\"}[5m])", - "legendFormat": "Rejections/sec" - } - ] - } - ] - } -} -``` - ---- - -## Production Readiness Checklist - -### Pre-Deployment - -- [ ] **Database migrations applied** - ```bash - cargo sqlx migrate run - psql -U foxhunt -c "\dt" | grep broker_orders - ``` - -- [ ] **FIX credentials verified** (Phase 2) - ```bash - telnet fix.cqg.com - # Should connect successfully - ``` - -- [ ] **Secrets encrypted** - ```bash - kubectl get secret broker-gateway-secrets -n foxhunt -o yaml | grep FIX_PASSWORD - # Should be base64 encoded - ``` - -- [ ] **Resource limits configured** - ```bash - kubectl describe pod broker-gateway-0 -n foxhunt | grep -A 5 Limits - ``` - -- [ ] **Health checks configured** - ```bash - kubectl describe pod broker-gateway-0 -n foxhunt | grep -A 10 Liveness - ``` - -- [ ] **Monitoring enabled** - ```bash - curl http://prometheus:9090/api/v1/targets | jq '.data.activeTargets[] | select(.labels.job == "broker-gateway")' - ``` - -- [ ] **Alerts configured** - ```bash - curl http://prometheus:9090/api/v1/rules | jq '.data.groups[] | select(.name == "broker-gateway")' - ``` - -### Post-Deployment - -- [ ] **Service health check passing** - ```bash - curl http://broker-gateway.foxhunt.svc.cluster.local:8086/health - # {"healthy": true, "message": "..."} - ``` - -- [ ] **gRPC endpoint accessible** - ```bash - grpcurl -plaintext broker-gateway.foxhunt.svc.cluster.local:50056 list - ``` - -- [ ] **FIX session active** (Phase 2) - ```bash - grpcurl -plaintext broker-gateway:50056 \ - broker_gateway.BrokerGatewayService/GetSessionStatus | jq .state - # Expected: "ACTIVE" (3) - ``` - -- [ ] **Metrics endpoint accessible** - ```bash - curl http://broker-gateway:9096/metrics | grep broker_orders_total - ``` - -- [ ] **Submit test order** - ```bash - grpcurl -plaintext -d '{ - "symbol": "ES", - "side": 1, - "quantity": 1, - "order_type": 1, - "account_id": "TEST_ACCT" - }' broker-gateway:50056 broker_gateway.BrokerGatewayService/RouteOrder - ``` - -- [ ] **Verify logs** - ```bash - kubectl logs broker-gateway-0 -n foxhunt --tail=100 | grep "Order submitted" - ``` - ---- - -## Rollback Procedure - -### Kubernetes Rollback - -```bash -# 1. Check rollout history -kubectl rollout history statefulset/broker-gateway -n foxhunt - -# Output: -# REVISION CHANGE-CAUSE -# 1 -# 2 Update to v0.1.1 - -# 2. Rollback to previous version -kubectl rollout undo statefulset/broker-gateway -n foxhunt - -# 3. Rollback to specific revision -kubectl rollout undo statefulset/broker-gateway -n foxhunt --to-revision=1 - -# 4. Monitor rollback -kubectl rollout status statefulset/broker-gateway -n foxhunt - -# 5. Verify health -kubectl get pods -n foxhunt -l app=broker-gateway -curl http://broker-gateway:8086/health -``` - -### Docker Rollback - -```bash -# 1. Stop current container -docker stop broker-gateway - -# 2. Start previous version -docker run -d \ - --name broker-gateway \ - --network foxhunt-network \ - -p 50056:50056 -p 8086:8086 -p 9096:9096 \ - -e DATABASE_URL=... \ - -e REDIS_URL=... \ - foxhunt/broker-gateway:v0.0.9 # Previous version - -# 3. Verify health -curl http://localhost:8086/health -``` - -### Database Rollback (if schema changed) - -```bash -# 1. Backup current database -pg_dump -U foxhunt foxhunt > /backup/foxhunt_pre_rollback.sql - -# 2. Rollback migrations -cargo sqlx migrate revert - -# 3. Verify schema -psql -U foxhunt -c "\d broker_orders" - -# 4. Restore data (if needed) -psql -U foxhunt foxhunt < /backup/foxhunt_backup.sql -``` - ---- - -## Scaling Guidelines - -### Horizontal Scaling - -```bash -# Scale StatefulSet manually -kubectl scale statefulset broker-gateway --replicas=5 -n foxhunt - -# Monitor scaling -kubectl get pods -n foxhunt -l app=broker-gateway -w - -# Verify load distribution (requires load balancer) -for i in {1..100}; do - grpcurl -plaintext broker-gateway:50056 \ - broker_gateway.BrokerGatewayService/HealthCheck -done -``` - -**Scaling Triggers**: - -| Metric | Scale Up Threshold | Scale Down Threshold | -|--------|-------------------|---------------------| -| CPU | >70% for 5 min | <30% for 10 min | -| Memory | >80% for 5 min | <40% for 10 min | -| Order Queue Size | >500 orders | <50 orders | -| P99 Latency | >10ms for 5 min | <2ms for 10 min | - -### Vertical Scaling - -```yaml -# Update resource limits -resources: - requests: - cpu: 1000m # Was 500m - memory: 1Gi # Was 512Mi - limits: - cpu: 4000m # Was 2000m - memory: 4Gi # Was 2Gi -``` - -```bash -# Apply changes -kubectl apply -f 4-statefulset.yaml - -# Rolling restart -kubectl rollout restart statefulset/broker-gateway -n foxhunt -``` - -### Database Scaling - -```bash -# Increase connection pool size -export DATABASE_MAX_CONNECTIONS=20 # Was 10 - -# Vertical scaling (PostgreSQL) -# - Increase shared_buffers (25% of RAM) -# - Increase max_connections (100+) -# - Enable connection pooling (PgBouncer) - -# Connection pooling with PgBouncer -docker run -d \ - --name pgbouncer \ - -e DB_HOST=postgres \ - -e DB_USER=foxhunt \ - -e DB_PASSWORD=foxhunt_dev_password \ - -e POOL_MODE=transaction \ - -e MAX_CLIENT_CONN=1000 \ - -e DEFAULT_POOL_SIZE=20 \ - -p 6432:6432 \ - edoburu/pgbouncer - -# Update DATABASE_URL -export DATABASE_URL=postgresql://foxhunt:foxhunt_dev_password@pgbouncer:6432/foxhunt -``` - ---- - -## Appendix: Example .env.production - -```bash -# Service configuration -GRPC_PORT=50056 -HEALTH_PORT=8086 -METRICS_PORT=9096 - -# Database -DATABASE_URL=postgresql://foxhunt:STRONG_PASSWORD@postgres.foxhunt.svc.cluster.local:5432/foxhunt -DATABASE_MAX_CONNECTIONS=20 - -# Redis -REDIS_URL=redis://redis.foxhunt.svc.cluster.local:6379 - -# Logging -RUST_LOG=info,broker_gateway_service=debug -LOG_FORMAT=json - -# FIX session (Phase 2) -FIX_SENDER_COMP_ID=FOXHUNT_CLIENT -FIX_TARGET_COMP_ID=CQG -FIX_USERNAME=your_cqg_username -FIX_PASSWORD=your_cqg_password -FIX_HOST=fix.cqg.com -FIX_PORT=xxxx -FIX_HEARTBEAT_INTERVAL=30 -FIX_RECONNECT_DELAY=5 -FIX_MAX_RETRIES=3 - -# Performance tuning -ORDER_QUEUE_SIZE=1000 -EXECUTION_STREAM_BUFFER=100 -``` - ---- - -## Support - -For deployment issues: -- **Slack**: #devops-support -- **Email**: devops@foxhunt.trading -- **Documentation**: https://wiki.foxhunt.trading/deployment diff --git a/services/broker_gateway_service/docs/TROUBLESHOOTING.md b/services/broker_gateway_service/docs/TROUBLESHOOTING.md deleted file mode 100644 index 11ab42862..000000000 --- a/services/broker_gateway_service/docs/TROUBLESHOOTING.md +++ /dev/null @@ -1,806 +0,0 @@ -# Broker Gateway Service Troubleshooting Guide - -**Version**: 0.1.0 -**Last Updated**: 2025-01-09 - -## Table of Contents - -- [Common Issues](#common-issues) - - [FIX Session Issues](#fix-session-issues) - - [Order Submission Issues](#order-submission-issues) - - [Position Discrepancies](#position-discrepancies) - - [Performance Issues](#performance-issues) -- [Debugging Tools](#debugging-tools) -- [Recovery Procedures](#recovery-procedures) -- [Log Analysis](#log-analysis) -- [Metrics and Monitoring](#metrics-and-monitoring) -- [Contact Information](#contact-information) - ---- - -## Common Issues - -### FIX Session Issues - -#### Issue: FIX Session Stuck in LOGGING_IN State - -**Symptoms**: -- Session state shows `LOGGING_IN` for >30 seconds -- No `Logon` response received from CQG -- Orders fail with `FAILED_PRECONDITION` - -**Diagnosis**: - -```bash -# 1. Check session status -grpcurl -plaintext localhost:50056 \ - broker_gateway.BrokerGatewayService/GetSessionStatus - -# Expected output (healthy): -# { -# "session_id": "FOXHUNT-CQG", -# "state": "ACTIVE", -# "sender_seq_num": "523", -# "target_seq_num": "498" -# } - -# 2. Check logs for Logon message -tail -f /var/log/broker_gateway/service.log | grep "Logon" - -# 3. Verify FIX credentials -echo $FIX_USERNAME $FIX_PASSWORD -``` - -**Common Causes**: - -| Cause | Fix | -|-------|-----| -| Invalid credentials | Verify `FIX_USERNAME` and `FIX_PASSWORD` env vars | -| Network firewall | Check CQG FIX port (TCP) is open | -| Wrong `SenderCompID` | Verify `FIX_SENDER_COMP_ID=FOXHUNT_CLIENT` | -| Wrong `TargetCompID` | Verify `FIX_TARGET_COMP_ID=CQG` | -| Sequence number mismatch | Reset sequences (see Recovery Procedures) | - -**Resolution**: - -```bash -# 1. Verify credentials -curl -X POST http://localhost:8086/admin/verify-credentials - -# 2. Reset FIX session -curl -X POST http://localhost:8086/admin/reset-session - -# 3. Restart service -systemctl restart broker-gateway - -# 4. Monitor reconnection -tail -f /var/log/broker_gateway/service.log -``` - ---- - -#### Issue: Heartbeat Timeout - -**Symptoms**: -- Session state transitions to `DISCONNECTED` -- Logs show: `No heartbeat received for 60s` -- Automatic reconnection attempts - -**Diagnosis**: - -```bash -# Check last heartbeat timestamp -grpcurl -plaintext localhost:50056 \ - broker_gateway.BrokerGatewayService/GetSessionStatus | jq .heartbeat_rtt_ms - -# Expected: < 100ms -# Actual: 0 (no heartbeat received) - -# Check network connectivity -ping fix.cqg.com -traceroute fix.cqg.com -``` - -**Common Causes**: - -1. **Network partition**: CQG unreachable -2. **CQG downtime**: Broker maintenance window -3. **Firewall dropped connection**: Idle timeout -4. **TCP keep-alive disabled**: Connection silently closed - -**Resolution**: - -```bash -# 1. Verify TCP keep-alive settings -sysctl net.ipv4.tcp_keepalive_time -# Expected: 300 (5 minutes) - -# 2. Force reconnect -curl -X POST http://localhost:8086/admin/reconnect - -# 3. Check CQG status page -curl https://status.cqg.com/api/v2/summary.json | jq . -``` - ---- - -#### Issue: Sequence Number Gap Detected - -**Symptoms**: -- Logs show: `Sequence gap: received 523, expected 498` -- FIX sends `ResendRequest` (MsgType=2) -- Session paused waiting for gap fill - -**Diagnosis**: - -```bash -# 1. Check current sequence numbers -grpcurl -plaintext localhost:50056 \ - broker_gateway.BrokerGatewayService/GetSessionStatus - -# Output: -# { -# "sender_seq_num": "497", # Our outgoing -# "target_seq_num": "498" # Expected incoming -# } - -# 2. Query database for persisted sequences -psql -U foxhunt -c "SELECT * FROM fix_sessions WHERE session_id = 'FOXHUNT-CQG';" - -# 3. Check for missing messages -grep "MsgSeqNum=49[0-9]" /var/log/broker_gateway/fix_messages.log -``` - -**Common Causes**: - -1. **Service restart mid-session**: Lost in-memory messages -2. **Database sequence desync**: Persisted sequence < actual -3. **CQG sequence reset**: Broker restarted without notification -4. **Duplicate message processing**: Same sequence processed twice - -**Resolution**: - -```bash -# Option 1: Request gap fill (automatic) -# Service sends ResendRequest, CQG sends SequenceReset or replays messages -# Wait 30-60 seconds for recovery - -# Option 2: Manual sequence sync -curl -X POST http://localhost:8086/admin/sync-sequences \ - -H "Content-Type: application/json" \ - -d '{"sender_seq": 497, "target_seq": 523}' - -# Option 3: Full reset (last resort - loses session state) -curl -X POST http://localhost:8086/admin/reset-session \ - -d '{"reset_sequences": true}' -``` - ---- - -### Order Submission Issues - -#### Issue: Order Rejected with "Insufficient Margin" - -**Symptoms**: -- `RouteOrder` succeeds, but `ExecutionReport` shows `ExecType=REJECTED` -- Text field: "Insufficient margin" -- Order status: `REJECTED` - -**Diagnosis**: - -```bash -# 1. Check account state -grpcurl -plaintext -d '{"account_id": "ACCT_001"}' localhost:50056 \ - broker_gateway.BrokerGatewayService/GetAccountState - -# Output: -# { -# "margin_available": "5000.00", # Only $5K available -# "buying_power": "20000.00" # 4x leverage -# } - -# 2. Calculate required margin for order -# ES margin: ~$13,000 per contract -# Order: 10 contracts = $130,000 required -# Available: $20,000 -# Result: REJECTED - -# 3. Query rejected orders -psql -U foxhunt -c " -SELECT client_order_id, symbol, quantity, status -FROM broker_orders -WHERE status = 'REJECTED' -AND account_id = 'ACCT_001' -ORDER BY created_at DESC -LIMIT 10; -" -``` - -**Resolution**: - -```bash -# Option 1: Reduce order size -# Submit 1 contract instead of 10 - -# Option 2: Close existing positions to free margin -grpcurl -plaintext -d '{"account_id": "ACCT_001"}' localhost:50056 \ - broker_gateway.BrokerGatewayService/GetPositions - -# Option 3: Add funds to account (contact broker) -# CQG account management: https://www.cqg.com/ -``` - ---- - -#### Issue: Order Stuck in PENDING_SUBMIT - -**Symptoms**: -- Order created in database -- Status: `PENDING_SUBMIT` -- No FIX `NewOrderSingle` sent -- No `ExecutionReport` received - -**Diagnosis**: - -```bash -# 1. Check FIX session state -grpcurl -plaintext localhost:50056 \ - broker_gateway.BrokerGatewayService/GetSessionStatus - -# If state != ACTIVE, orders are queued - -# 2. Check order queue size -curl http://localhost:9096/metrics | grep broker_order_queue_size -# broker_order_queue_size 47 # 47 orders queued - -# 3. Query pending orders -psql -U foxhunt -c " -SELECT client_order_id, symbol, quantity, created_at -FROM broker_orders -WHERE status = 'PENDING_SUBMIT' -AND created_at > NOW() - INTERVAL '10 minutes' -ORDER BY created_at; -" -``` - -**Common Causes**: - -1. **FIX session down**: Orders queued until session active -2. **Order queue full**: Backpressure applied (max 1000) -3. **Database transaction deadlock**: Order insert blocked -4. **Service shutting down**: Graceful shutdown in progress - -**Resolution**: - -```bash -# 1. Check FIX session health -curl http://localhost:8086/health | jq .checks.fix_session - -# 2. If session down, reconnect -curl -X POST http://localhost:8086/admin/reconnect - -# 3. Monitor queue drain -watch -n 1 'curl -s http://localhost:9096/metrics | grep broker_order_queue_size' - -# 4. If stuck after 5 minutes, restart service -systemctl restart broker-gateway -``` - ---- - -### Position Discrepancies - -#### Issue: Database Position != Broker Position - -**Symptoms**: -- `GetPositions` returns 10 ES contracts -- CQG shows 5 ES contracts (via web UI) -- Reconciliation alert triggered - -**Diagnosis**: - -```bash -# 1. Run manual reconciliation -curl -X POST http://localhost:8086/admin/reconcile-positions \ - -d '{"account_id": "ACCT_001"}' - -# Output: -# { -# "discrepancies": [ -# { -# "symbol": "ES", -# "broker_qty": 5.0, -# "db_qty": 10.0, -# "trading_qty": 10.0, -# "delta": -5.0 -# } -# ] -# } - -# 2. Query execution history -psql -U foxhunt -c " -SELECT execution_id, symbol, side, last_qty, last_price, transact_time -FROM broker_executions -WHERE symbol = 'ES' -AND account_id = 'ACCT_001' -ORDER BY transact_time DESC -LIMIT 20; -" - -# 3. Check for duplicate executions -psql -U foxhunt -c " -SELECT execution_id, COUNT(*) as count -FROM broker_executions -WHERE symbol = 'ES' -GROUP BY execution_id -HAVING COUNT(*) > 1; -" -``` - -**Common Causes**: - -| Cause | Fix | -|-------|-----| -| Duplicate `ExecutionReport` | Delete duplicate from database | -| Missed `ExecutionReport` | Insert phantom execution | -| Manual position adjustment (CQG web UI) | Accept broker as source of truth | -| Database rollback | Replay executions from FIX logs | - -**Resolution**: - -```bash -# 1. Accept broker position as source of truth -curl -X POST http://localhost:8086/admin/force-sync-position \ - -H "Content-Type: application/json" \ - -d '{ - "account_id": "ACCT_001", - "symbol": "ES", - "quantity": 5.0, - "source": "broker" - }' - -# 2. Verify sync -grpcurl -plaintext -d '{"account_id": "ACCT_001", "symbol": "ES"}' \ - localhost:50056 broker_gateway.BrokerGatewayService/GetPositions - -# 3. Notify Trading Service -curl -X POST http://localhost:50055/api/sync-position \ - -d '{"symbol": "ES", "quantity": 5.0}' -``` - ---- - -### Performance Issues - -#### Issue: High Order Submission Latency (>100ms) - -**Symptoms**: -- `RouteOrder` P99 latency: 150ms (target: <5ms) -- Grafana dashboard shows latency spike -- Users complain of slow order execution - -**Diagnosis**: - -```bash -# 1. Check database connection pool -curl http://localhost:9096/metrics | grep db_pool - -# Output: -# db_pool_active_connections 10 # All connections used -# db_pool_idle_connections 0 # No idle connections -# db_pool_wait_time_ms 45 # 45ms wait for connection - -# 2. Check database query performance -psql -U foxhunt -c " -SELECT query, mean_exec_time, calls -FROM pg_stat_statements -WHERE query LIKE '%broker_orders%' -ORDER BY mean_exec_time DESC -LIMIT 10; -" - -# 3. Check FIX session send queue -curl http://localhost:9096/metrics | grep fix_send_queue_size -# fix_send_queue_size 234 # 234 messages queued -``` - -**Common Causes**: - -1. **Database connection exhaustion**: Increase `max_connections` -2. **Slow database query**: Missing index on `client_order_id` -3. **FIX send queue backlog**: Broker slow to acknowledge -4. **High CPU usage**: Service overloaded - -**Resolution**: - -```bash -# 1. Increase database pool size -export DATABASE_MAX_CONNECTIONS=20 # Was 10 -systemctl restart broker-gateway - -# 2. Add missing index (if needed) -psql -U foxhunt -c " -CREATE INDEX CONCURRENTLY idx_broker_orders_account_status -ON broker_orders(account_id, status); -" - -# 3. Monitor improvements -watch -n 1 'curl -s http://localhost:9096/metrics | grep broker_order_submit_latency_seconds_bucket' -``` - ---- - -## Debugging Tools - -### 1. gRPC Health Probe - -```bash -# Install grpc-health-probe -curl -Lo /usr/local/bin/grpc-health-probe \ - https://github.com/grpc-ecosystem/grpc-health-probe/releases/download/v0.4.19/grpc-health-probe-linux-amd64 -chmod +x /usr/local/bin/grpc-health-probe - -# Check service health -grpc-health-probe -addr=localhost:50056 -# status: SERVING - -# Check specific service -grpc-health-probe -addr=localhost:50056 \ - -service=broker_gateway.BrokerGatewayService -``` - -### 2. grpcurl (gRPC CLI) - -```bash -# List services -grpcurl -plaintext localhost:50056 list - -# List methods -grpcurl -plaintext localhost:50056 list broker_gateway.BrokerGatewayService - -# Call method with JSON -grpcurl -plaintext \ - -d '{"account_id": "ACCT_001"}' \ - localhost:50056 \ - broker_gateway.BrokerGatewayService/GetAccountState -``` - -### 3. Prometheus Metrics - -```bash -# All metrics -curl http://localhost:9096/metrics - -# Filter by prefix -curl http://localhost:9096/metrics | grep broker_ - -# Key metrics: -# - broker_orders_total{status="SUBMITTED"} -# - broker_executions_total{exec_type="FILL"} -# - fix_session_state -# - broker_order_submit_latency_seconds -``` - -### 4. Database Queries - -```sql --- Active orders (not filled/cancelled/rejected) -SELECT client_order_id, symbol, side, quantity, status, created_at -FROM broker_orders -WHERE status IN ('PENDING_SUBMIT', 'SUBMITTED', 'PARTIALLY_FILLED') -ORDER BY created_at DESC; - --- Recent executions (last hour) -SELECT execution_id, symbol, side, last_qty, last_price, transact_time -FROM broker_executions -WHERE transact_time > NOW() - INTERVAL '1 hour' -ORDER BY transact_time DESC; - --- Position summary (calculated from executions) -SELECT - symbol, - SUM(CASE WHEN side = 'BUY' THEN last_qty ELSE -last_qty END) as net_qty, - COUNT(*) as trade_count -FROM broker_executions -WHERE exec_type = 'TRADE' -GROUP BY symbol; - --- Session state -SELECT session_id, state, sender_seq, target_seq, last_heartbeat, connected_at -FROM fix_sessions -WHERE session_id = 'FOXHUNT-CQG'; -``` - -### 5. Log Tailing - -```bash -# Service logs (structured JSON) -tail -f /var/log/broker_gateway/service.log | jq . - -# FIX message logs (raw protocol) -tail -f /var/log/broker_gateway/fix_messages.log - -# Error logs only -tail -f /var/log/broker_gateway/service.log | jq 'select(.level == "ERROR")' - -# Filter by order ID -tail -f /var/log/broker_gateway/service.log | jq 'select(.client_order_id == "550e8400-...")' -``` - ---- - -## Recovery Procedures - -### Procedure 1: Reset FIX Session - -**When to use**: Sequence number desync, session stuck, unrecoverable error - -**Steps**: - -```bash -# 1. Graceful logout -curl -X POST http://localhost:8086/admin/logout - -# 2. Wait 5 seconds -sleep 5 - -# 3. Reset session state -curl -X POST http://localhost:8086/admin/reset-session \ - -H "Content-Type: application/json" \ - -d '{ - "reset_sequences": true, - "clear_queue": false - }' - -# 4. Reconnect -curl -X POST http://localhost:8086/admin/reconnect - -# 5. Verify session active -grpcurl -plaintext localhost:50056 \ - broker_gateway.BrokerGatewayService/GetSessionStatus -``` - -**Expected duration**: 10-30 seconds - ---- - -### Procedure 2: Position Reconciliation - -**When to use**: Position discrepancy alert, after system outage - -**Steps**: - -```bash -# 1. Run reconciliation -curl -X POST http://localhost:8086/admin/reconcile-positions \ - -d '{"account_id": "ACCT_001"}' | jq . - -# 2. Review discrepancies -# Output shows broker_qty vs db_qty for each symbol - -# 3. Accept broker as source of truth -curl -X POST http://localhost:8086/admin/sync-all-positions \ - -d '{"account_id": "ACCT_001", "source": "broker"}' - -# 4. Notify Trading Service -curl -X POST http://localhost:50055/api/force-sync - -# 5. Verify sync -grpcurl -plaintext -d '{"account_id": "ACCT_001"}' \ - localhost:50056 broker_gateway.BrokerGatewayService/GetPositions -``` - ---- - -### Procedure 3: Database Recovery - -**When to use**: Database corruption, failed migration - -**Steps**: - -```bash -# 1. Stop service -systemctl stop broker-gateway - -# 2. Backup database -pg_dump -U foxhunt -t broker_orders -t broker_executions -t fix_sessions \ - foxhunt > /backup/broker_gateway_$(date +%Y%m%d_%H%M%S).sql - -# 3. Run migrations -cd /opt/foxhunt -cargo sqlx migrate run - -# 4. Verify schema -psql -U foxhunt -c "\d broker_orders" -psql -U foxhunt -c "\d broker_executions" - -# 5. Restore data (if needed) -psql -U foxhunt foxhunt < /backup/broker_gateway_20250109_120000.sql - -# 6. Start service -systemctl start broker-gateway - -# 7. Verify health -curl http://localhost:8086/health -``` - ---- - -## Log Analysis - -### Typical Log Flow (Successful Order) - -```json -# 1. Order received -{ - "timestamp": "2025-01-09T12:00:00.123Z", - "level": "INFO", - "message": "RouteOrder called", - "symbol": "ES", - "side": "BUY", - "quantity": 10, - "order_type": "MARKET" -} - -# 2. Validation passed -{ - "timestamp": "2025-01-09T12:00:00.124Z", - "level": "INFO", - "message": "Order validation passed", - "client_order_id": "550e8400-e29b-41d4-a716-446655440000" -} - -# 3. Saved to database -{ - "timestamp": "2025-01-09T12:00:00.126Z", - "level": "INFO", - "message": "Order saved to database", - "client_order_id": "550e8400-...", - "status": "PENDING_SUBMIT" -} - -# 4. FIX message sent (Phase 2) -{ - "timestamp": "2025-01-09T12:00:00.128Z", - "level": "DEBUG", - "message": "FIX NewOrderSingle sent", - "client_order_id": "550e8400-...", - "msg_seq_num": 523 -} - -# 5. ExecutionReport received (Phase 2) -{ - "timestamp": "2025-01-09T12:00:00.350Z", - "level": "INFO", - "message": "ExecutionReport received", - "execution_id": "EXEC_789", - "broker_order_id": "BROKER_123", - "client_order_id": "550e8400-...", - "exec_type": "NEW", - "order_status": "SUBMITTED" -} -``` - -### Error Log Examples - -```json -# Validation error -{ - "timestamp": "2025-01-09T12:05:00.123Z", - "level": "ERROR", - "message": "Order validation failed", - "error": "Price is required for LIMIT orders", - "symbol": "ES", - "order_type": "LIMIT", - "price": null -} - -# Database error -{ - "timestamp": "2025-01-09T12:10:00.456Z", - "level": "ERROR", - "message": "Failed to insert order into database", - "error": "connection pool exhausted", - "client_order_id": "550e8400-..." -} - -# FIX session error (Phase 2) -{ - "timestamp": "2025-01-09T12:15:00.789Z", - "level": "ERROR", - "message": "FIX session heartbeat timeout", - "last_heartbeat_received": "2025-01-09T12:14:00.000Z", - "timeout_threshold_secs": 60 -} -``` - ---- - -## Metrics and Monitoring - -### Key Metrics to Watch - -```promql -# Order submission rate (orders/sec) -rate(broker_orders_total[1m]) - -# Order rejection rate -rate(broker_orders_total{status="REJECTED"}[5m]) -/ rate(broker_orders_total[5m]) - -# Execution fill rate -rate(broker_executions_total{exec_type="FILL"}[1m]) - -# FIX session uptime (1 = active, 0 = down) -fix_session_state == 3 - -# Order submission latency P99 -histogram_quantile(0.99, broker_order_submit_latency_seconds_bucket) - -# Position reconciliation discrepancies -position_reconciliation_discrepancies > 0 -``` - -### Grafana Dashboard Panels - -1. **Order Flow**: - - Orders submitted/sec - - Executions received/sec - - Order status breakdown (pie chart) - -2. **FIX Session Health**: - - Session state (gauge: 0-4) - - Heartbeat RTT (line chart) - - Sequence numbers (sender vs target) - -3. **Latency**: - - RouteOrder P50/P99/P99.9 - - CancelOrder P50/P99/P99.9 - - FIX send latency - -4. **Errors**: - - Rejection rate - - Sequence gaps/hour - - Position discrepancies - ---- - -## Contact Information - -### On-Call Escalation - -| Tier | Contact | Response SLA | -|------|---------|--------------| -| **L1 Support** | Slack: #broker-gateway-support | 15 minutes | -| **L2 Engineering** | PagerDuty: Broker Gateway Team | 30 minutes | -| **L3 Infrastructure** | Phone: +1-XXX-XXX-XXXX | 1 hour | - -### External Contacts - -| Vendor | Contact | Purpose | -|--------|---------|---------| -| **CQG Support** | support@cqg.com | FIX session issues, broker downtime | -| **AMP Futures** | Phone: 1-800-560-1640 | Account issues, margin calls | -| **AWS Support** | Console | Database, Redis, networking | - -### Documentation - -- **Internal Wiki**: https://wiki.foxhunt.trading/broker-gateway -- **CQG FIX Docs**: https://partners.cqg.com/api-resources/fix-api -- **Runbook**: https://runbook.foxhunt.trading/broker-gateway - ---- - -## Appendix: Common Error Messages - -| Error Message | Cause | Fix | -|---------------|-------|-----| -| `Symbol is required` | Empty symbol field | Provide valid symbol (ES, NQ, etc.) | -| `Quantity must be positive` | quantity <= 0 | Set quantity > 0 | -| `Price is required for LIMIT orders` | Missing price for LIMIT | Add price field | -| `Order not found` | Invalid client_order_id | Verify order ID is correct | -| `Order cannot be cancelled (status: FILLED)` | Cancel filled order | Cannot cancel filled orders | -| `FIX session not active` | Session down | Wait for reconnect or manually reset | -| `Database connection failed` | PostgreSQL down | Check database health | -| `Insufficient margin` | Low account balance | Reduce order size or add funds | -| `Sequence gap detected` | Missing FIX messages | Wait for ResendRequest/gap fill | diff --git a/services/ml_training_service/HYPERPARAMETER_TUNING.md b/services/ml_training_service/HYPERPARAMETER_TUNING.md deleted file mode 100644 index 8635cebe1..000000000 --- a/services/ml_training_service/HYPERPARAMETER_TUNING.md +++ /dev/null @@ -1,956 +0,0 @@ -# Hyperparameter Tuning with Optuna - -## Overview - -The ML Training Service includes an Optuna-based hyperparameter tuning subsystem that optimizes model parameters for maximum Sharpe ratio. The tuner runs as a Python subprocess, coordinating with the Rust gRPC service to train models with sampled hyperparameters. - -## Architecture - -``` -โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” -โ”‚ ML Training Service (Rust gRPC) โ”‚ -โ”‚ Port 50054 โ”‚ -โ”‚ โ”‚ -โ”‚ StartTuningJob() โ†’ spawn Python subprocess โ”‚ -โ”‚ โ”‚ -โ”‚ TrainModel() โ† called by Optuna for each trial โ”‚ -โ”‚ โ”‚ -โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ - โ†• gRPC -โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” -โ”‚ hyperparameter_tuner.py (Python Subprocess) โ”‚ -โ”‚ โ”‚ -โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ -โ”‚ โ”‚ Optuna Study (JournalStorage) โ”‚ โ”‚ -โ”‚ โ”‚ โ”‚ โ”‚ -โ”‚ โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ โ”‚ -โ”‚ โ”‚ โ”‚ Trial 1: Sample params โ”‚ โ”‚ โ”‚ -โ”‚ โ”‚ โ”‚ โ†’ Call TrainModel gRPC โ”‚ โ”‚ โ”‚ -โ”‚ โ”‚ โ”‚ โ†’ Receive Sharpe ratio โ”‚ โ”‚ โ”‚ -โ”‚ โ”‚ โ”‚ โ†’ Report to Optuna โ”‚ โ”‚ โ”‚ -โ”‚ โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ โ”‚ -โ”‚ โ”‚ โ”‚ โ”‚ -โ”‚ โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ โ”‚ -โ”‚ โ”‚ โ”‚ Trial 2: Sample params โ”‚ โ”‚ โ”‚ -โ”‚ โ”‚ โ”‚ โ†’ Call TrainModel gRPC โ”‚ โ”‚ โ”‚ -โ”‚ โ”‚ โ”‚ โ†’ Receive Sharpe ratio โ”‚ โ”‚ โ”‚ -โ”‚ โ”‚ โ”‚ โ†’ Report to Optuna โ”‚ โ”‚ โ”‚ -โ”‚ โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ โ”‚ -โ”‚ โ”‚ โ”‚ โ”‚ -โ”‚ โ”‚ ... (sequential trials) โ”‚ โ”‚ -โ”‚ โ”‚ โ”‚ โ”‚ -โ”‚ โ”‚ MedianPruner: Early stopping โ”‚ โ”‚ -โ”‚ โ”‚ TPE Sampler: Smart search โ”‚ โ”‚ -โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ -โ”‚ โ”‚ -โ”‚ GPU Monitor (pynvml): Check VRAM availability โ”‚ -โ”‚ JournalStorage: Persist state to MinIO after each trial โ”‚ -โ”‚ SIGTERM Handler: Graceful shutdown โ”‚ -โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ - โ†“ -โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” -โ”‚ MinIO Storage (Crash Recovery) โ”‚ -โ”‚ โ”‚ -โ”‚ /studies/study_.log โ† JournalStorage file โ”‚ -โ”‚ โ”‚ -โ”‚ Format: Optuna SQLite-free journaling system โ”‚ -โ”‚ Benefits: File-based, crash recovery, no DB dependency โ”‚ -โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ -``` - -## Key Design Decisions - -### 1. Sequential Trials (n_jobs=1) -**Critical for GPU memory safety**: -- RTX 3050 Ti has 4GB VRAM -- Training large models (TLOB, MAMBA-2, TFT) can consume 2-3GB -- Parallel trials would cause OOM errors -- Sequential execution ensures only one model in GPU memory at a time - -### 2. JournalStorage (File-Based Persistence) -**Crash recovery without database**: -- Optuna 3.0+ JournalStorage writes study state to file after each trial -- No SQLite/PostgreSQL dependency -- Survives process crashes, kernel panics, power loss -- Study can be resumed from any point - -### 3. MedianPruner (Early Stopping) -**Efficient search space exploration (30-50% time savings)**: -- Prunes trials that perform worse than median of previous trials -- After warmup period (10 epochs), compares intermediate Sharpe ratios -- If trial's Sharpe < median โ†’ prune (stop trial early) -- Configurable via `tuning_config.yaml`: - - `n_startup_trials: 5` - No pruning for first 5 trials (establish baseline) - - `n_warmup_steps: 10` - Wait 10 epochs before pruning (allow convergence) - - `interval_steps: 5` - Check every 5 epochs (balance overhead vs responsiveness) -- Expected impact: 30-50% reduction in total tuning time - -### 4. GPU Memory Monitoring (pynvml) -**Direct NVIDIA library access**: -- Uses `pynvml` Python bindings (NOT `nvidia-smi` subprocess) -- Zero overhead compared to subprocess spawning -- Real-time VRAM usage tracking -- Automatic trial skipping if insufficient memory - -### 5. Graceful Shutdown (SIGTERM Handler) -**Production-ready lifecycle management**: -- Captures SIGTERM/SIGINT signals -- Completes current trial before shutdown -- Persists study state to storage -- Clean gRPC channel closure - -## Setup - -### 1. Install Python Dependencies - -```bash -cd services/ml_training_service -pip3 install -r requirements-tuner.txt -``` - -**Dependencies**: -- `optuna>=3.0.0` - Hyperparameter optimization framework -- `grpcio>=1.50.0` - gRPC client -- `PyYAML>=6.0` - Config file parsing -- `nvidia-ml-py3>=7.352.0` - GPU monitoring - -### 2. Generate Python gRPC Stubs - -```bash -cd services/ml_training_service -./scripts/generate_python_proto.sh -``` - -This generates: -- `proto/ml_training_pb2.py` - Proto message classes -- `proto/ml_training_pb2_grpc.py` - gRPC client stubs - -### 3. Configure Search Spaces - -Edit `tuning_config.yaml` to define hyperparameter search spaces: - -```yaml -global: - optimization_direction: maximize # maximize sharpe_ratio - pruning_enabled: true - median_pruner: - n_startup_trials: 5 # No pruning for first 5 trials (establish baseline) - n_warmup_steps: 10 # Wait 10 epochs before starting to prune - interval_steps: 5 # Check for pruning every 5 epochs - sampler: TPE # Tree-structured Parzen Estimator - -models: - TLOB: - epochs: - type: int - low: 10 - high: 100 - step: 10 - learning_rate: - type: float - low: 0.00001 - high: 0.01 - log: true - # ... more parameters -``` - -**Search Space Types**: -- `int`: Integer range with step -- `float`: Continuous range (optionally log-scale) -- `categorical`: Discrete choices (including booleans) - -## Usage - -### Command-Line Invocation - -```bash -python3 hyperparameter_tuner.py \ - --job-id tuning_job_12345 \ - --model-type TLOB \ - --num-trials 50 \ - --config tuning_config.yaml \ - --data-source-json '{"file_path": "/data/btc_usd_2024.parquet", "start_time": 1704067200, "end_time": 1704672000}' \ - --use-gpu \ - --storage-path /minio/studies/study_12345.log \ - --grpc-host localhost \ - --grpc-port 50054 -``` - -**Arguments**: -- `--job-id`: Unique identifier for this tuning job (generated by Rust service) -- `--model-type`: Model to optimize (`TLOB`, `MAMBA_2`, `DQN`, `PPO`, `LIQUID`, `TFT`) -- `--num-trials`: Number of hyperparameter combinations to test -- `--config`: Path to tuning config YAML (default: `tuning_config.yaml`) -- `--data-source-json`: Training data specification as JSON -- `--use-gpu`: Enable GPU acceleration (requires CUDA) -- `--storage-path`: Path to Optuna JournalStorage file (for crash recovery) -- `--grpc-host`: ML Training Service host (default: `localhost`) -- `--grpc-port`: ML Training Service gRPC port (default: `50054`) - -### Data Source JSON Format - -```json -{ - "file_path": "/data/btc_usd_2024.parquet", - "start_time": 1704067200, - "end_time": 1704672000 -} -``` - -Or for database query: - -```json -{ - "historical_db_query": "SELECT * FROM market_data WHERE symbol='BTC/USD'", - "start_time": 1704067200, - "end_time": 1704672000 -} -``` - -### Rust Service Integration - -The Rust service spawns the tuner subprocess when `StartTuningJob` is called: - -```rust -// In services/ml_training_service/src/service.rs - -use tokio::process::Command; - -pub async fn start_tuning_job( - &self, - request: StartTuningJobRequest, -) -> Result { - let job_id = Uuid::new_v4().to_string(); - let storage_path = format!("/minio/studies/study_{}.log", job_id); - - let data_source_json = serde_json::to_string(&request.data_source)?; - - let mut cmd = Command::new("python3"); - cmd.arg("hyperparameter_tuner.py") - .arg("--job-id").arg(&job_id) - .arg("--model-type").arg(&request.model_type) - .arg("--num-trials").arg(request.num_trials.to_string()) - .arg("--config").arg(&request.config_path) - .arg("--data-source-json").arg(&data_source_json) - .arg("--storage-path").arg(&storage_path) - .arg("--grpc-host").arg("localhost") - .arg("--grpc-port").arg("50054"); - - if request.use_gpu { - cmd.arg("--use-gpu"); - } - - // Spawn subprocess (non-blocking) - let child = cmd.spawn()?; - - // Store child process for monitoring/cleanup - self.tuning_jobs.insert(job_id.clone(), child); - - Ok(StartTuningJobResponse { - job_id, - status: TuningJobStatus::TuningRunning, - message: "Tuning job started successfully".to_string(), - }) -} -``` - -## Optimization Process - -### Trial Lifecycle - -1. **Parameter Sampling**: - - Optuna's TPE sampler suggests hyperparameter values - - Values sampled from search spaces in `tuning_config.yaml` - - Early trials: Random exploration - - Later trials: Focused on promising regions - -2. **Model Training (via gRPC)**: - - Python subprocess calls `TrainModel` gRPC endpoint - - Rust service trains model with sampled parameters - - Returns `TrainModelResponse` with Sharpe ratio and metrics - -3. **Objective Evaluation**: - - Sharpe ratio is optimization objective (higher is better) - - Additional metrics stored as trial attributes (loss, duration, etc.) - -4. **Pruning Decision** (MedianPruner): - - Compare trial performance to median of previous trials - - If worse than median, prune (early stopping) - - Saves GPU time by skipping unpromising trials - -5. **State Persistence**: - - JournalStorage writes trial result to file - - Study can be resumed if process crashes - -6. **Next Trial**: - - Repeat until `num_trials` reached or shutdown signal - -### Optimization Algorithm (TPE) - -**Tree-structured Parzen Estimator**: -- Bayesian optimization method -- Models p(x|y) where: - - x = hyperparameters - - y = objective value (Sharpe ratio) -- Builds two distributions: - - l(x): Good hyperparameters (high Sharpe) - - g(x): Bad hyperparameters (low Sharpe) -- Samples from l(x) to maximize expected improvement - -**Advantages**: -- Sample-efficient (fewer trials needed) -- Handles categorical/conditional parameters -- Parallelizable (though we use n_jobs=1) - -## Crash Recovery - -### Automatic Resumption - -If the tuner subprocess crashes (OOM, kernel panic, power loss): - -```bash -# Original command -python3 hyperparameter_tuner.py \ - --job-id tuning_job_12345 \ - --model-type TLOB \ - --num-trials 50 \ - --storage-path /minio/studies/study_12345.log \ - # ... other args - -# Re-run with SAME arguments (especially --job-id and --storage-path) -# Optuna will load existing study and continue from last completed trial -``` - -**JournalStorage Guarantees**: -- Study state persisted after each trial -- No data loss on crash -- Automatic deduplication of trial numbers -- Preserves TPE sampler state - -### Monitoring Recovery - -Check study state manually: - -```python -import optuna -from optuna.storages import JournalStorage, JournalFileStorage - -storage = JournalStorage(JournalFileStorage("/minio/studies/study_12345.log")) -study = optuna.load_study(study_name="study_tuning_job_12345", storage=storage) - -print(f"Completed trials: {len(study.trials)}") -print(f"Best Sharpe ratio: {study.best_value:.4f}") -print(f"Best params: {study.best_params}") -``` - -## GPU Memory Management - -### Memory Monitoring - -The tuner monitors GPU memory before each trial: - -```python -gpu_monitor = GPUMonitor() - -# Check if 2GB VRAM available -if not gpu_monitor.check_memory_available(required_gb=2.0): - logger.warning("Insufficient GPU memory, pruning trial") - raise optuna.TrialPruned() -``` - -### Memory Usage Tracking - -After each trial, GPU utilization is logged: - -``` -[2025-10-13 15:30:45] [INFO] Trial 10: GPU memory: 2.34/4.00 GB (58.5%) -``` - -### OOM Prevention Strategies - -1. **Sequential trials** (n_jobs=1): Only one model in VRAM at a time -2. **Pre-trial memory check**: Skip trial if <2GB available -3. **Post-trial cleanup**: Rely on Rust service to free GPU memory -4. **Batch size tuning**: Search space includes batch_size (adjust for VRAM) - -## Graceful Shutdown - -### Signal Handling - -```python -import signal - -shutdown_requested = False - -def signal_handler(signum, frame): - global shutdown_requested - logger.info("Shutdown requested, completing current trial...") - shutdown_requested = True - -signal.signal(signal.SIGTERM, signal_handler) -signal.signal(signal.SIGINT, signal_handler) -``` - -### Shutdown Process - -1. Capture SIGTERM/SIGINT -2. Set `shutdown_requested = True` -3. Complete current trial (don't interrupt gRPC call) -4. Persist final study state to storage -5. Close gRPC channel -6. Exit with code 0 (success) - -### Orchestration from Rust - -```rust -// Stop tuning job gracefully -pub async fn stop_tuning_job(&self, job_id: &str) -> Result<()> { - if let Some(mut child) = self.tuning_jobs.remove(job_id) { - // Send SIGTERM (NOT SIGKILL) - child.kill().await?; - - // Wait for graceful shutdown (max 30 seconds) - match tokio::time::timeout( - Duration::from_secs(30), - child.wait() - ).await { - Ok(Ok(status)) => { - log::info!("Tuning job {} exited: {}", job_id, status); - } - Ok(Err(e)) => { - log::error!("Failed to wait for tuning job {}: {}", job_id, e); - } - Err(_) => { - log::warn!("Tuning job {} did not exit in 30s, force killing", job_id); - let _ = child.kill().await; - } - } - } - Ok(()) -} -``` - -## Tuning Results - -### Best Parameters - -After tuning completes, retrieve best hyperparameters via `GetTuningJobStatus`: - -```rust -let response = client.get_tuning_job_status(GetTuningJobStatusRequest { - job_id: "tuning_job_12345".to_string(), -}).await?; - -println!("Best Sharpe ratio: {}", response.best_metrics["sharpe_ratio"]); -println!("Best hyperparameters:"); -for (param, value) in response.best_params { - println!(" {}: {}", param, value); -} -``` - -### Trial History - -All trial results are available in `GetTuningJobStatusResponse.trial_history`: - -```rust -for trial in response.trial_history { - println!("Trial {}: Sharpe={:.4f}, Duration={}s, State={:?}", - trial.trial_number, - trial.objective_value, - trial.completed_at - trial.started_at, - trial.state - ); - - for (param, value) in trial.params { - println!(" {}: {}", param, value); - } -} -``` - -### Visualization - -Export study for analysis: - -```python -import optuna -from optuna.visualization import plot_optimization_history, plot_param_importances - -storage = JournalStorage(JournalFileStorage("/minio/studies/study_12345.log")) -study = optuna.load_study(study_name="study_tuning_job_12345", storage=storage) - -# Optimization progress -fig = plot_optimization_history(study) -fig.write_html("optimization_history.html") - -# Parameter importance -fig = plot_param_importances(study) -fig.write_html("param_importances.html") -``` - -## Performance Considerations - -### Trial Duration - -Typical trial durations (TLOB on RTX 3050 Ti): -- Epochs=10, batch_size=128: ~2-3 minutes -- Epochs=50, batch_size=64: ~8-12 minutes -- Epochs=100, batch_size=32: ~20-30 minutes - -**50 trials with 10 epochs each: ~2-3 hours** - -### Tuning Job Sizing - -Recommended trial counts by model complexity: - -| Model Type | Simple Search | Thorough Search | -|------------|---------------|-----------------| -| LIQUID | 20-30 trials | 50-100 trials | -| DQN | 30-50 trials | 100-200 trials | -| PPO | 30-50 trials | 100-200 trials | -| TLOB | 50-100 trials | 200-500 trials | -| MAMBA_2 | 50-100 trials | 200-500 trials | -| TFT | 50-100 trials | 200-500 trials | - -### Search Space Sizing - -Number of unique hyperparameter combinations: - -- TLOB: ~10,000+ combinations (9 parameters) -- MAMBA_2: ~5,000+ combinations (8 parameters) -- DQN: ~100,000+ combinations (12 parameters, many categorical) - -**TPE sampler converges in 50-200 trials for most search spaces** - -## Troubleshooting - -### Common Issues - -#### 1. gRPC Connection Failed - -``` -ERROR: Failed to connect to gRPC service: -``` - -**Solution**: Ensure ML Training Service is running: - -```bash -cargo run -p ml_training_service -# Check health -grpc_health_probe -addr=localhost:50054 -``` - -#### 2. GPU Out of Memory - -``` -WARNING: Insufficient GPU memory, pruning trial -``` - -**Solutions**: -- Reduce `batch_size` in search space -- Lower `hidden_dim` / `num_layers` upper bounds -- Check for leaked VRAM (restart service) - -#### 3. Proto Import Error - -``` -ImportError: cannot import name 'ml_training_pb2' from 'proto' -``` - -**Solution**: Regenerate Python stubs: - -```bash -./scripts/generate_python_proto.sh -``` - -#### 4. Tuning Config Not Found - -``` -FileNotFoundError: [Errno 2] No such file or directory: 'tuning_config.yaml' -``` - -**Solution**: Ensure config file exists in working directory or specify absolute path: - -```bash -python3 hyperparameter_tuner.py \ - --config /home/user/foxhunt/services/ml_training_service/tuning_config.yaml \ - # ... other args -``` - -#### 5. Study Load Error - -``` -ERROR: Study 'study_tuning_job_12345' not found in storage -``` - -**Solution**: Check storage path matches original run: - -```bash -# List studies in storage -ls -la /minio/studies/*.log -``` - -### Debug Logging - -Enable verbose logging: - -```bash -export PYTHONUNBUFFERED=1 # Disable output buffering -export LOG_LEVEL=DEBUG - -python3 hyperparameter_tuner.py --config ... 2>&1 | tee tuning.log -``` - -## Testing - -### Unit Tests - -```bash -cd services/ml_training_service -python3 -m pytest tests/test_hyperparameter_tuner.py -v -``` - -### Integration Test - -```bash -# Start ML Training Service -cargo run -p ml_training_service & - -# Wait for service to be ready -sleep 5 - -# Run tuner with small trial count -python3 hyperparameter_tuner.py \ - --job-id test_job \ - --model-type LIQUID \ - --num-trials 5 \ - --config tuning_config.yaml \ - --data-source-json '{"file_path": "/tmp/test_data.parquet", "start_time": 0, "end_time": 1}' \ - --storage-path /tmp/test_study.log \ - --grpc-host localhost \ - --grpc-port 50054 - -# Check results -python3 -c " -import optuna -from optuna.storages import JournalStorage, JournalFileStorage -storage = JournalStorage(JournalFileStorage('/tmp/test_study.log')) -study = optuna.load_study(study_name='study_test_job', storage=storage) -print(f'Trials completed: {len(study.trials)}') -print(f'Best value: {study.best_value:.4f}') -" -``` - -## MedianPruner Configuration Details - -### How MedianPruner Works - -**Pruning Logic**: -1. **Baseline Phase** (Trials 0-4): - - First 5 trials complete fully without pruning - - Establishes median performance baseline - - Required for meaningful comparison - -2. **Warmup Phase** (Epochs 0-9): - - Each trial trains for 10 epochs without pruning checks - - Allows models to initialize and start converging - - Prevents premature pruning during early training instability - -3. **Pruning Phase** (Epochs 10, 15, 20, 25, ...): - - At each `interval_steps` checkpoint (every 5 epochs): - - Report intermediate Sharpe ratio via `trial.report(sharpe, step=epoch)` - - Compare to **median** of completed trials at same step - - If `current_sharpe < median_sharpe` โ†’ **PRUNE** (raise `optuna.TrialPruned()`) - - Else continue training to next checkpoint - -### Example Pruning Scenario - -``` -Trial 0-4 (Baseline): All complete 100 epochs - Trial 0: Sharpe = 1.2 - Trial 1: Sharpe = 1.5 - Trial 2: Sharpe = 1.8 - Trial 3: Sharpe = 1.4 - Trial 4: Sharpe = 1.6 - -Median Sharpe @ epoch 100: 1.5 - -Trial 5 (First prunable): - Epoch 10: Sharpe = 0.8 (< 1.5) โ†’ PRUNED โœ‚๏ธ - Time saved: 90 epochs ร— 2 min/epoch = 180 minutes (3 hours) - -Trial 6 (Promising): - Epoch 10: Sharpe = 1.6 (โ‰ฅ 1.5) โ†’ Continue โœ“ - Epoch 15: Sharpe = 1.7 (โ‰ฅ 1.5) โ†’ Continue โœ“ - ... completes 100 epochs โ†’ Sharpe = 1.9 (new best!) - -Trial 7 (Mediocre): - Epoch 10: Sharpe = 1.5 (= median) โ†’ Continue โœ“ - Epoch 15: Sharpe = 1.4 (< 1.5) โ†’ PRUNED โœ‚๏ธ - Time saved: 85 epochs ร— 2 min/epoch = 170 minutes -``` - -### Parameter Tuning Guidelines - -**n_startup_trials** (Default: 5): -- Too low (1-2): Unstable baseline, random pruning -- Too high (10+): Wastes time, delays pruning benefits -- Recommended: 5-10 trials (10-20% of total trials) - -**n_warmup_steps** (Default: 10): -- Too low (0-5): Premature pruning during initialization -- Too high (20+): Pruning happens too late to save time -- Recommended: 10-20% of total epochs - - 10 epochs for 100-epoch training - - 5 epochs for 50-epoch training - - 20 epochs for 200-epoch training - -**interval_steps** (Default: 5): -- Too low (1): High overhead from frequent checks -- Too high (20+): Delayed pruning, less time savings -- Recommended: 3-5 checks per trial on average - - 5 epochs for 100-epoch training (20 checks) - - 3 epochs for 50-epoch training (17 checks) - - 10 epochs for 200-epoch training (20 checks) - -### Expected Performance Impact - -**Time Savings Calculation**: - -Assumptions: -- 50 trials total -- 100 epochs per trial -- 2 minutes per epoch -- 40% of trials pruned (industry benchmark) -- Average pruning at 35% completion - -Without MedianPruner: -``` -50 trials ร— 100 epochs ร— 2 min/epoch = 10,000 minutes (166.7 hours) -``` - -With MedianPruner: -``` -5 trials (baseline) ร— 100 epochs ร— 2 min = 1,000 min -25 trials (complete) ร— 100 epochs ร— 2 min = 5,000 min -20 trials (pruned) ร— 35 epochs ร— 2 min = 1,400 min -โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ -Total: 7,400 minutes (123.3 hours) - -Time saved: 2,600 minutes (43.3 hours, 26% reduction) -``` - -**Industry Benchmarks**: -- MedianPruner typically saves 30-50% of tuning time -- Effectiveness depends on: - - Search space quality (bad ranges โ†’ more pruning) - - Model convergence speed (faster โ†’ earlier pruning) - - Objective metric stability (noisy โ†’ less pruning) - -### Current Implementation Limitations - -**No Intra-Trial Early Stopping**: - -The current gRPC interface (`TrainModel`) returns only **final** metrics after all epochs complete. This means: - -โœ… **What Works**: -- MedianPruner compares **final** Sharpe ratios across completed trials -- Future trials benefit from knowing which parameter ranges are unpromising -- Inter-trial pruning: "If trial 5 had Sharpe=0.8, don't sample similar params again" - -โŒ **What Doesn't Work**: -- Cannot prune a trial mid-training (e.g., after 15 epochs of 100) -- Full 30-50% time savings not achievable -- Current savings limited to ~10-15% from faster convergence decisions - -**Why This Limitation Exists**: -- `TrainModel` RPC is unary: single request โ†’ single response -- Rust service trains for ALL epochs before returning -- No mechanism for Python client to interrupt training mid-epoch - -**Example of Missing Capability**: -```python -# What we WANT to do (but can't with current interface): -for epoch in range(100): - sharpe = train_one_epoch() - trial.report(sharpe, step=epoch) - - if trial.should_prune(): # Check after each epoch - logger.info(f"Pruning at epoch {epoch}") - return early # Stop training, save time -``` - -### Future Enhancement: Streaming gRPC - -**To achieve full 30-50% time savings**, the gRPC interface needs to support intermediate reporting: - -**Option 1: Streaming Response** (Recommended): - -```protobuf -// ml_training.proto -rpc TrainModelWithProgress(TrainModelRequest) returns (stream TrainingProgress); - -message TrainingProgress { - uint32 current_epoch = 1; - float current_sharpe = 2; - bool training_complete = 3; - map metrics = 4; -} -``` - -**Option 2: Callback URL**: -- Python provides HTTP callback endpoint -- Rust POSTs progress every N epochs -- Python checks pruning and sends stop signal if needed - -**Option 3: Status Polling**: -- Python polls Rust service every 30 seconds -- Rust exposes `GetTrainingProgress(trial_id)` endpoint -- Python checks pruning and calls `StopTraining(trial_id)` if needed - -**Recommended**: Option 1 (Streaming) for efficiency and real-time updates. - -**Estimated Effort**: 6-8 hours to implement streaming gRPC support. - -### Testing MedianPruner - -**Unit Test** (Mock gRPC responses): - -```python -def test_median_pruner_prunes_low_performers(): - """Verify MedianPruner prunes trials below median.""" - - mock_responses = [ - {"sharpe_ratio": 1.2}, # Trial 0 (baseline) - {"sharpe_ratio": 1.5}, # Trial 1 (baseline) - {"sharpe_ratio": 1.8}, # Trial 2 (baseline) - {"sharpe_ratio": 1.4}, # Trial 3 (baseline) - {"sharpe_ratio": 1.6}, # Trial 4 (baseline) - {"sharpe_ratio": 0.5}, # Trial 5 (should prune - << median) - ] - - with patch_grpc_client(mock_responses): - study = optuna.create_study( - direction="maximize", - pruner=MedianPruner( - n_startup_trials=5, - n_warmup_steps=10, - interval_steps=5 - ) - ) - - study.optimize(objective_function, n_trials=6) - - # Verify trial 5 was pruned - assert study.trials[5].state == optuna.trial.TrialState.PRUNED - assert study.trials[5].value == 0.5 - - # Verify baseline trials completed - for i in range(5): - assert study.trials[i].state == optuna.trial.TrialState.COMPLETE -``` - -**Integration Test** (Real training with small dataset): - -```bash -# Run small tuning job to verify pruning -python3 hyperparameter_tuner.py \ - --job-id test_pruning \ - --model-type LIQUID \ - --num-trials 10 \ - --config tuning_config.yaml \ - --data-source-json '{"file_path": "test_data/small_sample.parquet"}' \ - --storage-path /tmp/test_pruning_study.log - -# Verify pruning statistics -python3 -c " -import optuna -from optuna.storages import JournalStorage, JournalFileStorage - -storage = JournalStorage(JournalFileStorage('/tmp/test_pruning_study.log')) -study = optuna.load_study(study_name='study_test_pruning', storage=storage) - -pruned = [t for t in study.trials if t.state == optuna.trial.TrialState.PRUNED] -completed = [t for t in study.trials if t.state == optuna.trial.TrialState.COMPLETE] - -print(f'Completed: {len(completed)}, Pruned: {len(pruned)}') -print(f'Pruning rate: {len(pruned)/(len(pruned)+len(completed))*100:.1f}%') - -assert len(pruned) > 0, 'No trials were pruned! MedianPruner may not be working.' -assert len(completed) >= 5, 'Not enough baseline trials completed!' -" -``` - -### Monitoring Pruning Behavior - -**View Study Statistics**: - -```python -import optuna -from optuna.storages import JournalStorage, JournalFileStorage - -storage = JournalStorage(JournalFileStorage('/minio/studies/study_.log')) -study = optuna.load_study(study_name='study_', storage=storage) - -# Trial state distribution -states = [t.state.name for t in study.trials] -from collections import Counter -print(Counter(states)) -# Output: {'COMPLETE': 30, 'PRUNED': 18, 'FAIL': 2} - -# Pruning statistics -completed = [t for t in study.trials if t.state == optuna.trial.TrialState.COMPLETE] -pruned = [t for t in study.trials if t.state == optuna.trial.TrialState.PRUNED] - -print(f"Pruning rate: {len(pruned)/(len(pruned)+len(completed))*100:.1f}%") -print(f"Median Sharpe (completed): {np.median([t.value for t in completed]):.4f}") -print(f"Median Sharpe (pruned): {np.median([t.value for t in pruned]):.4f}") - -# Time savings estimate -avg_epoch_time = 2.0 # minutes -total_epochs = 100 -pruning_epochs = 35 # average pruning point - -time_saved = len(pruned) * (total_epochs - pruning_epochs) * avg_epoch_time -print(f"Estimated time saved: {time_saved:.0f} minutes ({time_saved/60:.1f} hours)") -``` - -**Debug Pruning Decisions**: - -```python -# Analyze why a specific trial was pruned -trial_num = 5 -trial = study.trials[trial_num] - -if trial.state == optuna.trial.TrialState.PRUNED: - print(f"Trial {trial_num} was pruned") - print(f"Final Sharpe ratio: {trial.value:.4f}") - - # Get intermediate values reported - for step, value in trial.intermediate_values.items(): - print(f" Epoch {step}: Sharpe = {value:.4f}") - - # Compare to median at pruning point - prune_step = max(trial.intermediate_values.keys()) - prior_trials = [t for t in study.trials[:trial_num] - if t.state == optuna.trial.TrialState.COMPLETE] - - if prior_trials: - median_sharpe = np.median([t.value for t in prior_trials]) - print(f"\nMedian Sharpe (trials 0-{trial_num-1}): {median_sharpe:.4f}") - print(f"Trial {trial_num} Sharpe at pruning: {trial.value:.4f}") - print(f"Difference: {trial.value - median_sharpe:.4f} ({'below' if trial.value < median_sharpe else 'above'} median)") -``` - -## References - -- [Optuna Documentation](https://optuna.readthedocs.io/) -- [Optuna JournalStorage](https://optuna.readthedocs.io/en/stable/reference/storages.html#optuna.storages.JournalStorage) -- [TPE Sampler](https://optuna.readthedocs.io/en/stable/reference/samplers/generated/optuna.samplers.TPESampler.html) -- [MedianPruner](https://optuna.readthedocs.io/en/stable/reference/pruners/generated/optuna.pruners.MedianPruner.html) -- [MedianPruner Tutorial](https://optuna.readthedocs.io/en/stable/tutorial/10_key_features/003_efficient_optimization_algorithms.html#pruning) -- [pynvml Documentation](https://pypi.org/project/nvidia-ml-py3/) -- [gRPC Streaming Guide](https://grpc.io/docs/what-is-grpc/core-concepts/#server-streaming-rpc) - -## License - -See project root LICENSE file. diff --git a/services/ml_training_service/TRIAL_EXECUTOR_USAGE.md b/services/ml_training_service/TRIAL_EXECUTOR_USAGE.md deleted file mode 100644 index 2f4a0d525..000000000 --- a/services/ml_training_service/TRIAL_EXECUTOR_USAGE.md +++ /dev/null @@ -1,499 +0,0 @@ -# Trial Executor Usage Guide - -## Overview - -The `TrialExecutor` provides a fixed-size thread pool for concurrent execution of Optuna hyperparameter tuning trials with automatic GPU resource management. It ensures efficient GPU utilization while preventing resource contention. - -## Architecture - -``` -โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” -โ”‚ Optuna Subprocess โ”‚ -โ”‚ (Python hyperparameter_tuner.py) โ”‚ -โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ - โ”‚ Submit trial - โ–ผ -โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” -โ”‚ TrialExecutor โ”‚ -โ”‚ (Fixed Pool) โ”‚ -โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ -โ”‚ โ”‚ Worker 0 โ”‚ โ”‚ Worker 1 โ”‚ โ”‚ Worker N โ”‚ โ”‚ -โ”‚ โ”‚ GPU 0 โ”‚ โ”‚ GPU 1 โ”‚ โ”‚ GPU N โ”‚ โ”‚ -โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ -โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ - โ”‚ โ”‚ โ”‚ - โ–ผ โ–ผ โ–ผ - โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” - โ”‚ ML Training Service (gRPC) โ”‚ - โ”‚ TrainModel RPC โ”‚ - โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ -``` - -### Key Components - -1. **Fixed Pool Size**: Number of workers = Number of GPUs -2. **Worker Thread**: Tokio task that processes trials sequentially -3. **GPU Assignment**: Each worker owns a dedicated GPU (via CUDA_VISIBLE_DEVICES) -4. **Trial Queue**: MPSC channel for FIFO trial scheduling -5. **Resource Tracking**: Per-worker statistics and monitoring - -## GPU Detection - -The executor automatically detects available GPUs: - -```rust -// Automatic GPU detection (reads CUDA_VISIBLE_DEVICES) -let executor = TrialExecutor::new("http://localhost:50054".to_string()).await?; -// Pool size = GPU count (defaults to 1 if detection fails) - -// Manual pool size override -let executor = TrialExecutor::with_pool_size("http://localhost:50054".to_string(), 2).await?; -``` - -### GPU Detection Logic - -1. Check `CUDA_VISIBLE_DEVICES` environment variable -2. Parse comma-separated device IDs (e.g., "0,1,2" โ†’ 3 GPUs) -3. Default to 1 GPU if not set or parsing fails - -**Example Environment Variables**: -```bash -# Single GPU (RTX 3050 Ti) -export CUDA_VISIBLE_DEVICES=0 - -# Multi-GPU system -export CUDA_VISIBLE_DEVICES=0,1,2,3 - -# Let executor detect automatically -unset CUDA_VISIBLE_DEVICES # Defaults to 1 -``` - -## Basic Usage - -### 1. Initialize Executor - -```rust -use ml_training_service::trial_executor::TrialExecutor; -use anyhow::Result; - -#[tokio::main] -async fn main() -> Result<()> { - // Create executor with automatic GPU detection - let executor = TrialExecutor::new("http://localhost:50054".to_string()).await?; - - println!("Pool size: {}", executor.pool_size()); - - // ... use executor ... - - // Graceful shutdown (60-second timeout) - executor.shutdown().await?; - Ok(()) -} -``` - -### 2. Submit Trials - -```rust -use std::collections::HashMap; -use ml_training_service::service::proto::DataSource; - -// Prepare trial parameters -let mut hyperparameters = HashMap::new(); -hyperparameters.insert("learning_rate".to_string(), 0.001); -hyperparameters.insert("batch_size".to_string(), 64.0); -hyperparameters.insert("hidden_size".to_string(), 128.0); - -let data_source = DataSource { - // ... data source configuration ... -}; - -// Submit trial (returns oneshot receiver for result) -let result_rx = executor - .submit_trial( - "trial_001".to_string(), - "TLOB".to_string(), - hyperparameters, - data_source, - true, // use_gpu - ) - .await?; - -// Wait for result -let result = result_rx.await??; -println!("Trial complete: success={}, sharpe_ratio={:.4}", - result.success, result.sharpe_ratio); -``` - -### 3. Monitor Pool Statistics - -```rust -// Get current pool statistics -let stats = executor.get_stats().await; - -println!("Pool size: {}", stats.pool_size); -println!("Active trials: {}", stats.active_trials); -println!("Total completed: {}", stats.total_completed); -println!("Total failed: {}", stats.total_failed); - -// Per-worker statistics -for worker in &stats.worker_stats { - println!("Worker {} (GPU {}): completed={}, failed={}, oom_errors={}, busy={}", - worker.worker_id, worker.gpu_id, worker.trials_completed, - worker.trials_failed, worker.oom_errors, worker.is_busy); -} -``` - -## Integration with Optuna - -The trial executor is designed to be called from Optuna's objective function: - -### Python Optuna Integration - -```python -# hyperparameter_tuner.py - -import optuna -import grpc -from ml_training_service_pb2 import TrainModelRequest -from ml_training_service_pb2_grpc import MlTrainingServiceStub - -def objective(trial: optuna.Trial) -> float: - """Optuna objective function that calls ML Training Service via gRPC.""" - - # Sample hyperparameters - hyperparameters = { - "learning_rate": trial.suggest_float("learning_rate", 1e-5, 1e-2, log=True), - "batch_size": trial.suggest_int("batch_size", 16, 128, step=16), - "hidden_size": trial.suggest_int("hidden_size", 64, 512, step=64), - "dropout": trial.suggest_float("dropout", 0.1, 0.5), - } - - # Connect to ML Training Service - channel = grpc.insecure_channel("localhost:50054") - stub = MlTrainingServiceStub(channel) - - # Call TrainModel RPC (routed through TrialExecutor) - request = TrainModelRequest( - trial_id=str(trial.number), - model_type="TLOB", - hyperparameters=hyperparameters, - use_gpu=True, - # ... data source ... - ) - - response = stub.TrainModel(request) - - if not response.success: - raise optuna.TrialPruned(f"Training failed: {response.error_message}") - - # Return Sharpe ratio as objective (maximize) - return response.sharpe_ratio - -# Create study -study = optuna.create_study( - direction="maximize", - study_name="tlob_hyperparameter_tuning", -) - -# Run optimization -study.optimize(objective, n_trials=100) - -print(f"Best trial: {study.best_trial.number}") -print(f"Best Sharpe ratio: {study.best_value:.4f}") -print(f"Best params: {study.best_params}") -``` - -### Rust Service Integration - -```rust -use ml_training_service::trial_executor::TrialExecutor; -use ml_training_service::tuning_manager::TuningManager; -use std::sync::Arc; - -pub struct MLTrainingServiceImpl { - trial_executor: Arc, - tuning_manager: Arc, -} - -impl MLTrainingServiceImpl { - pub async fn new(grpc_endpoint: String) -> Result { - // Initialize trial executor - let trial_executor = Arc::new( - TrialExecutor::new(grpc_endpoint).await? - ); - - // Initialize tuning manager - let tuning_manager = Arc::new( - TuningManager::new( - "scripts/hyperparameter_tuner.py".to_string(), - "/tmp/tuning_jobs".to_string(), - ) - ); - - Ok(Self { - trial_executor, - tuning_manager, - }) - } - - // TrainModel RPC handler (called by Optuna) - async fn train_model( - &self, - request: TrainModelRequest, - ) -> Result { - // Submit trial to executor - let result_rx = self.trial_executor - .submit_trial( - request.trial_id, - request.model_type, - request.hyperparameters, - request.data_source.unwrap(), - request.use_gpu, - ) - .await?; - - // Wait for result - let result = result_rx.await??; - - Ok(TrainModelResponse { - success: result.success, - sharpe_ratio: result.sharpe_ratio, - training_loss: result.training_loss, - validation_metrics: result.validation_metrics, - error_message: result.error_message, - training_duration_seconds: result.training_duration_seconds, - }) - } -} -``` - -## Resource Management - -### Worker Lifecycle - -1. **Initialization**: Worker spawned with dedicated GPU ID -2. **Waiting**: Worker blocks on trial queue (1-second timeout) -3. **Execution**: Worker runs trial with CUDA_VISIBLE_DEVICES set -4. **Completion**: Worker updates statistics and returns result -5. **Shutdown**: Worker finishes current trial and exits - -### GPU Isolation - -Each worker sets `CUDA_VISIBLE_DEVICES` for its subprocess: - -```rust -// Worker 0 โ†’ GPU 0 -std::env::set_var("CUDA_VISIBLE_DEVICES", "0"); - -// Worker 1 โ†’ GPU 1 -std::env::set_var("CUDA_VISIBLE_DEVICES", "1"); -``` - -**Note**: This only affects subprocesses spawned by the TrainModel gRPC call, not the current process. - -### Graceful Shutdown - -```rust -// Shutdown with 60-second timeout -executor.shutdown().await?; - -// Shutdown behavior: -// 1. Cancel shutdown token (signals workers to stop) -// 2. Wait for current trials to finish (max 60s) -// 3. Force kill workers after timeout -``` - -## Error Handling - -### OOM (Out of Memory) Errors - -The executor tracks OOM errors per worker: - -```rust -let stats = executor.get_stats().await; -for worker in &stats.worker_stats { - if worker.oom_errors > 0 { - warn!("Worker {} experienced {} OOM errors", - worker.worker_id, worker.oom_errors); - } -} -``` - -### Worker Crash Recovery - -Workers are **not** automatically restarted on crash. The pool remains at reduced capacity until manual intervention. - -**Future Enhancement**: Add worker restart logic: -```rust -// TODO: Implement worker crash recovery -// - Detect worker exit via JoinHandle -// - Spawn replacement worker -// - Maintain pool_size workers -``` - -### Trial Timeout - -Trials timeout after 30 minutes: - -```rust -// Built into execute_trial_with_gpu -match timeout(Duration::from_secs(1800), client.train_model(request)).await { - Ok(Ok(resp)) => /* success */, - Ok(Err(e)) => /* gRPC error */, - Err(_) => /* timeout after 30 minutes */, -} -``` - -## Performance Tuning - -### Pool Size Recommendations - -| Hardware | Pool Size | Rationale | -|----------|-----------|-----------| -| Single GPU (RTX 3050 Ti) | 1 | Sequential trials (validated in research) | -| 2 GPUs | 2 | Parallel trials, 1 per GPU | -| 4 GPUs | 4 | Parallel trials, 1 per GPU | -| 8 GPUs | 8 | Parallel trials, 1 per GPU | - -**Rule of Thumb**: `pool_size = num_gpus` for maximum throughput without contention. - -### Memory Management - -- **Trial Isolation**: Each trial runs in separate subprocess with dedicated GPU -- **No Sharing**: Workers never share GPU memory -- **OOM Detection**: Tracked per worker, enables batch size reduction strategies - -### Monitoring - -Add Prometheus metrics (future enhancement): - -```rust -// TODO: Add Prometheus metrics -// - trial_executor_pool_size (gauge) -// - trial_executor_active_trials (gauge) -// - trial_executor_trials_completed_total (counter) -// - trial_executor_trials_failed_total (counter) -// - trial_executor_oom_errors_total (counter) -// - trial_executor_trial_duration_seconds (histogram) -``` - -## Testing - -### Unit Tests - -```bash -# Run trial executor tests -cargo test -p ml_training_service --test trial_executor_test - -# Run with logging -RUST_LOG=debug cargo test -p ml_training_service --test trial_executor_test -- --nocapture -``` - -### Integration Tests - -```bash -# Start ML Training Service -cargo run -p ml_training_service - -# Run Optuna hyperparameter tuning -python scripts/hyperparameter_tuner.py --model-type TLOB --num-trials 10 -``` - -### Load Testing - -```bash -# Simulate 100 concurrent trials -python scripts/optuna_load_test.py --num-trials 100 --num-workers 4 -``` - -## Configuration - -### Environment Variables - -```bash -# GPU configuration -export CUDA_VISIBLE_DEVICES=0,1,2,3 # Multi-GPU - -# gRPC endpoint -export ML_TRAINING_SERVICE_ENDPOINT=http://localhost:50054 - -# Logging -export RUST_LOG=info,ml_training_service::trial_executor=debug -``` - -### Runtime Configuration - -```rust -// Custom pool size (override GPU detection) -let executor = TrialExecutor::with_pool_size( - "http://localhost:50054".to_string(), - 2 // 2 workers even if 4 GPUs detected -).await?; -``` - -## Troubleshooting - -### Issue: Pool size is 1 but I have multiple GPUs - -**Solution**: Set `CUDA_VISIBLE_DEVICES` environment variable: -```bash -export CUDA_VISIBLE_DEVICES=0,1,2,3 -``` - -### Issue: Workers stuck in busy state - -**Symptoms**: `worker.is_busy = true` for extended period - -**Debugging**: -```bash -# Check if gRPC service is responding -grpcurl -plaintext localhost:50054 list - -# Check GPU utilization -nvidia-smi -``` - -### Issue: High OOM error rate - -**Solution**: Reduce batch size in hyperparameters: -```python -# Optuna objective function -hyperparameters = { - "batch_size": trial.suggest_int("batch_size", 8, 32, step=8), # Reduced range -} -``` - -### Issue: Shutdown timeout - -**Symptoms**: Shutdown takes 60+ seconds - -**Cause**: Workers still running trials - -**Solution**: Wait for active trials to complete before shutting down: -```rust -// Check active trials -let stats = executor.get_stats().await; -if stats.active_trials > 0 { - warn!("Waiting for {} active trials to complete", stats.active_trials); - tokio::time::sleep(Duration::from_secs(10)).await; -} - -executor.shutdown().await?; -``` - -## Future Enhancements - -1. **Worker Crash Recovery**: Automatically restart crashed workers -2. **Batch Size Reduction**: Retry OOM trials with reduced batch size -3. **Prometheus Metrics**: Real-time monitoring and alerting -4. **Trial Prioritization**: Priority queue instead of FIFO -5. **Multi-Node Distribution**: Distribute trials across multiple machines -6. **GPU Memory Profiling**: Track per-trial memory usage -7. **Adaptive Timeout**: Dynamic timeout based on model type and data size - -## References - -- **Research Validation**: Single GPU (RTX 3050 Ti) validated for sequential trial execution -- **gRPC Protocol**: `services/ml_training_service/proto/ml_training.proto` -- **Optuna Documentation**: https://optuna.readthedocs.io/ -- **CUDA Documentation**: https://docs.nvidia.com/cuda/ diff --git a/services/ml_training_service/tests/fixtures/README.md b/services/ml_training_service/tests/fixtures/README.md deleted file mode 100644 index 909ad9daa..000000000 --- a/services/ml_training_service/tests/fixtures/README.md +++ /dev/null @@ -1,81 +0,0 @@ -# Integration Test Fixtures - -This directory contains test data fixtures for integration testing. - -## Directory Structure - -``` -fixtures/ -โ”œโ”€โ”€ small_parquet_files/ # Small Parquet files for quick tests (100 bars each) -โ”œโ”€โ”€ invalid_parquet_files/ # Corrupted files for error testing -โ”œโ”€โ”€ job_configs/ # Valid and invalid job configurations -โ””โ”€โ”€ README.md -``` - -## Test Data Files - -### Small Parquet Files -- **Purpose**: Quick training tests without loading large datasets -- **Size**: ~100 OHLCV bars per file -- **Usage**: Real data integration tests - -Use the actual small Parquet files from `/test_data/` directory: -- `ES_FUT_small.parquet` -- `NQ_FUT_small.parquet` -- `6E_FUT_small.parquet` -- `ZN_FUT_small.parquet` - -### Invalid Parquet Files -- **Purpose**: Error handling validation -- **Types**: Corrupted headers, missing columns, invalid data types - -### Job Configurations - -#### Valid Configurations -- `valid_dqn.json`: DQN model with standard hyperparameters -- `valid_ppo.json`: PPO model configuration -- `valid_mamba2.json`: MAMBA-2 configuration -- `valid_tft.json`: TFT configuration - -#### Invalid Configurations -- `invalid_zero_epochs.json`: max_epochs = 0 (should fail validation) -- `invalid_negative_lr.json`: learning_rate < 0 (should fail) -- `invalid_huge_batch.json`: batch_size = 1000000 (resource error) - -## Creating New Fixtures - -### Generate Small Parquet Files - -```bash -# From project root -cargo run --example create_small_parquet_files -``` - -This creates 100-bar samples from real market data for fast testing. - -### Validate Fixtures - -```bash -# Run integration tests -cd services/ml_training_service -cargo test --test integration -- --test-threads=1 -``` - -## Usage in Tests - -```rust -use std::path::Path; - -// Load valid config -let config_path = Path::new("tests/fixtures/job_configs/valid_dqn.json"); -let config = std::fs::read_to_string(config_path)?; - -// Load test parquet -let parquet_path = Path::new("test_data/ES_FUT_small.parquet"); -``` - -## Maintenance - -- Update fixtures when adding new models or features -- Keep file sizes small (<100KB for quick CI/CD) -- Document any non-obvious test cases diff --git a/services/trading_agent_service/README_TLS.md b/services/trading_agent_service/README_TLS.md deleted file mode 100644 index 4062ce35c..000000000 --- a/services/trading_agent_service/README_TLS.md +++ /dev/null @@ -1,112 +0,0 @@ -# Trading Agent Service TLS Implementation - -## Quick Start - -### Enable TLS - -```bash -# Set environment variables -export TLS_ENABLED=true -export MTLS_ENABLED=true # Optional: enable mutual TLS - -# Start service -cargo run --bin trading_agent_service -``` - -### Generate Test Certificates - -```bash -# Run the automated setup script -./scripts/verify_tls.sh - -# Or manually: -cd /tmp/foxhunt/certs - -# 1. Generate CA -openssl genrsa -out ca.key 4096 -openssl req -new -x509 -days 365 -key ca.key \ - -out ca.crt \ - -subj "/CN=Foxhunt Trading Agent CA" - -# 2. Generate server certificate -openssl genrsa -out server.key 4096 -openssl req -new -key server.key \ - -out server.csr \ - -subj "/CN=trading-agent-service" - -# 3. Sign server certificate -openssl x509 -req -days 365 \ - -in server.csr \ - -CA ca.crt \ - -CAkey ca.key \ - -CAcreateserial \ - -out server.crt -``` - -## Configuration - -### Environment Variables - -| Variable | Default | Description | -|---|---|---| -| `TLS_ENABLED` | `false` | Enable/disable TLS | -| `MTLS_ENABLED` | `false` | Enable mutual TLS (client cert validation) | -| `TLS_CERT_PATH` | `/tmp/foxhunt/certs/server.crt` | Server certificate path | -| `TLS_KEY_PATH` | `/tmp/foxhunt/certs/server.key` | Server private key path | -| `TLS_CA_PATH` | `/tmp/foxhunt/certs/ca.crt` | CA certificate path (for mTLS) | - -## Testing - -```bash -# Run TLS tests (non-ignored) -cargo test --test tls_test - -# Run all tests including integration tests -cargo test --test tls_test -- --ignored - -# Verify TLS configuration -./scripts/verify_tls.sh -``` - -## Client Connection Example - -```rust -use tonic::transport::{Certificate, Channel, ClientTlsConfig, Identity}; - -async fn connect() -> Result { - let cert_pem = tokio::fs::read_to_string("/tmp/foxhunt/certs/client.crt").await?; - let key_pem = tokio::fs::read_to_string("/tmp/foxhunt/certs/client.key").await?; - let ca_pem = tokio::fs::read_to_string("/tmp/foxhunt/certs/ca.crt").await?; - - let tls = ClientTlsConfig::new() - .identity(Identity::from_pem(cert_pem, key_pem)) - .ca_certificate(Certificate::from_pem(ca_pem)) - .domain_name("trading-agent-service"); - - Channel::from_shared("https://localhost:50055")? - .tls_config(tls)? - .connect() - .await -} -``` - -## Documentation - -For complete documentation, see: -- `AGENT_S6_TLS_TRADING_AGENT_SERVICE_COMPLETE.md` - Full implementation guide -- `tests/tls_test.rs` - Test suite examples -- `scripts/verify_tls.sh` - Verification script - -## Support - -For issues or questions: -1. Check the verification script output: `./scripts/verify_tls.sh` -2. Verify certificates are valid: `openssl x509 -in /tmp/foxhunt/certs/server.crt -text -noout` -3. Check service logs for TLS-related errors - -## Security Notes - -- **Production**: Use proper certificate management (Vault, cert-manager, etc.) -- **Development**: Test certificates are fine for local testing -- **mTLS**: Enable for maximum security in production environments -- **TLS 1.3**: Modern protocol with improved security and performance diff --git a/services/trading_service/docs/ml_integration_design.md b/services/trading_service/docs/ml_integration_design.md deleted file mode 100644 index 89ab99478..000000000 --- a/services/trading_service/docs/ml_integration_design.md +++ /dev/null @@ -1,1011 +0,0 @@ -# ML Integration Design: Trading Service Adaptive Strategy - -**Mission**: Design ML inference engine integration with trading service using **TDD methodology** - -**Status**: Design Phase (Wave 10, Agent 10.9) - -**Date**: 2025-10-15 - ---- - -## Executive Summary - -This document outlines the integration of the ML inference engine (`RealMLInferenceEngine` from `ml/src/inference.rs`) with the trading service's strategy execution system. The integration will enable production-ready ML-powered trading signals while maintaining the existing rule-based strategy as a fallback. - -### Key Integration Points - -1. **Enhanced ML Service** (`services/trading_service/src/services/enhanced_ml.rs`) - Primary ML inference interface -2. **ML Strategy Engine** (`services/backtesting_service/src/ml_strategy_engine.rs`) - Strategy-level ML coordination -3. **Adaptive Strategy** (`adaptive-strategy/src/lib.rs`) - High-level strategy orchestration - -### TDD Philosophy - -**RED โ†’ GREEN โ†’ REFACTOR** - -1. **RED**: Write failing tests defining expected ML integration behavior -2. **GREEN**: Implement minimal code to pass tests -3. **REFACTOR**: Improve code quality while maintaining test coverage - ---- - -## Architecture Overview - -``` -โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” -โ”‚ Trading Service โ”‚ -โ”‚ โ”‚ -โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ -โ”‚ โ”‚ Enhanced ML Service (Primary Interface) โ”‚ โ”‚ -โ”‚ โ”‚ โ”‚ โ”‚ -โ”‚ โ”‚ โ€ข RealMLInferenceEngine (ml/src/inference.rs) โ”‚ โ”‚ -โ”‚ โ”‚ โ€ข 4 Production Models: DQN, PPO, MAMBA-2, TFT โ”‚ โ”‚ -โ”‚ โ”‚ โ€ข Feature extraction (256-dim UnifiedFinancialFeatures) โ”‚ โ”‚ -โ”‚ โ”‚ โ€ข Ensemble voting (confidence-weighted) โ”‚ โ”‚ -โ”‚ โ”‚ โ€ข Safety validation (MLSafetyManager) โ”‚ โ”‚ -โ”‚ โ”‚ โ€ข GPU acceleration (RTX 3050 Ti CUDA) โ”‚ โ”‚ -โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ -โ”‚ โ†“ โ”‚ -โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ -โ”‚ โ”‚ ML Strategy Executor (Strategy Layer) โ”‚ โ”‚ -โ”‚ โ”‚ โ”‚ โ”‚ -โ”‚ โ”‚ โ€ข Market data โ†’ ML predictions โ†’ Trading signals โ”‚ โ”‚ -โ”‚ โ”‚ โ€ข Position sizing (confidence-based) โ”‚ โ”‚ -โ”‚ โ”‚ โ€ข Risk validation (leverage, VaR, position limits) โ”‚ โ”‚ -โ”‚ โ”‚ โ€ข Performance tracking (Sharpe, accuracy, latency) โ”‚ โ”‚ -โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ -โ”‚ โ†“ โ”‚ -โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ -โ”‚ โ”‚ Trading Service gRPC Handler โ”‚ โ”‚ -โ”‚ โ”‚ โ”‚ โ”‚ -โ”‚ โ”‚ โ€ข Order submission (submit_order) โ”‚ โ”‚ -โ”‚ โ”‚ โ€ข Risk checks (kill switch, position limits) โ”‚ โ”‚ -โ”‚ โ”‚ โ€ข Order execution (via TradingRepository) โ”‚ โ”‚ -โ”‚ โ”‚ โ€ข Audit logging (event persistence) โ”‚ โ”‚ -โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ -โ”‚ โ”‚ -โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ -``` - ---- - -## Component Analysis - -### 1. ML Inference Engine (`ml/src/inference.rs`) - -**Current Implementation**: - -```rust -pub struct RealMLInferenceEngine { - config: RealInferenceConfig, - models: Arc>>, - safety_manager: Arc, - prediction_cache: Arc>>, - performance_metrics: Arc>, -} - -pub struct RealPredictionResult { - pub model_id: Uuid, - pub symbol: Symbol, - pub timestamp: DateTime, - pub prediction: Price, // Safe common::Price type - pub confidence: f64, // 0.0 to 1.0 - pub uncertainty: f64, // Prediction std dev - pub feature_importance: HashMap, - pub drift_score: f64, - pub inference_latency_us: u64, - pub lower_bound: Price, // Risk management bounds - pub upper_bound: Price, -} -``` - -**Key Features**: -- โœ… **4 Production Models**: DQN, PPO, MAMBA-2, TFT (trainable adapters ready) -- โœ… **GPU Acceleration**: RTX 3050 Ti CUDA support with CPU fallback -- โœ… **Safety Validation**: MLSafetyManager with NaN/Inf checks, drift detection -- โœ… **Prediction Caching**: 60-second TTL for sub-microsecond cache hits -- โœ… **Prometheus Metrics**: Latency, accuracy, confidence, drift, cache hits -- โœ… **Feature Extraction**: 256-dimensional UnifiedFinancialFeatures (OHLCV + technical indicators) - -**Performance Targets**: -- Inference latency: **<50ฮผs** (HFT requirement) -- Confidence threshold: **>0.7** (minimum for trading signals) -- Drift score: **<0.1** (model stability) -- GPU memory: **<1GB** (RTX 3050 Ti constraint) - ---- - -### 2. ML Strategy Engine (`services/backtesting_service/src/ml_strategy_engine.rs`) - -**Current Implementation**: - -```rust -pub struct MLPoweredStrategy { - name: String, - models: HashMap>, - feature_extractor: MLFeatureExtractor, - model_performance: HashMap, - confidence_based_sizing: bool, - min_confidence_threshold: f64, -} - -// Simplified execution (current) -impl StrategyExecutor for MLPoweredStrategy { - fn execute(&self, market_data: &MarketData, portfolio: &Portfolio, - parameters: &HashMap) -> Result>; -} -``` - -**Key Features**: -- โœ… **Feature Extraction**: Price momentum, moving averages, volatility, volume ratios -- โœ… **Ensemble Voting**: Confidence-weighted predictions from multiple models -- โœ… **Performance Tracking**: Accuracy, Sharpe ratio, latency per model -- โœ… **Confidence-Based Sizing**: Position size scales with prediction confidence -- โš ๏ธ **Simplified Models**: DQN/Transformer simulators (not production inference engine) - -**Integration Gap**: -- Currently uses `MLModelSimulator` trait (mock implementations) -- Needs integration with `RealMLInferenceEngine` for production -- Feature extraction duplicated (should use `UnifiedFinancialFeatures`) - ---- - -### 3. Adaptive Strategy (`adaptive-strategy/src/lib.rs`) - -**Current Implementation**: - -```rust -pub struct AdaptiveStrategy { - config: config::AdaptiveStrategyConfig, - ensemble: Arc>, - state: Arc>, -} - -impl AdaptiveStrategy { - pub async fn execute_strategy_cycle(&self) -> Result<()> { - // 1. Update market regime - // 2. Get ensemble predictions - // 3. Calculate position sizes - // 4. Execute trades - // 5. Update performance metrics - } -} -``` - -**Key Features**: -- โœ… **Regime Detection**: Market regime classification (trending, mean-reverting, volatile) -- โœ… **Ensemble Coordination**: Multi-model strategy orchestration -- โœ… **PostgreSQL Configuration**: Database-backed config with hot-reload -- โœ… **Performance Tracking**: Sharpe, drawdown, win rate, trade count -- โš ๏ธ **Stub Implementation**: Strategy cycle needs ML inference integration - ---- - -## Data Flow Design - -### Feature Engineering Pipeline - -``` -Market Data (OHLCV bars) - โ†“ -โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” -โ”‚ UnifiedFinancialFeatures::extract_ml_features() โ”‚ -โ”‚ (ml/src/features/unified.rs) โ”‚ -โ”‚ โ”‚ -โ”‚ โ€ข 5 OHLCV features (normalized) โ”‚ -โ”‚ โ€ข 10 Technical indicators (RSI, MACD, Bollinger, โ”‚ -โ”‚ ATR, EMA, volume ratios, price momentum) โ”‚ -โ”‚ โ€ข Time-based features (hour, day of week) โ”‚ -โ”‚ โ€ข Total: 256 dimensions (padded) โ”‚ -โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ - โ†“ -FeatureVector (Vec, length=256) - โ†“ -โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” -โ”‚ RealMLInferenceEngine::predict() โ”‚ -โ”‚ โ”‚ -โ”‚ 1. Validate features (256-dim check, finite values)โ”‚ -โ”‚ 2. Convert to tensor [1, 256] on GPU/CPU โ”‚ -โ”‚ 3. Forward pass through neural network โ”‚ -โ”‚ 4. Safety validation (NaN/Inf, drift, confidence) โ”‚ -โ”‚ 5. Return RealPredictionResult โ”‚ -โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ - โ†“ -RealPredictionResult (price prediction + metadata) - โ†“ -โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” -โ”‚ Ensemble Voting (confidence-weighted) โ”‚ -โ”‚ โ”‚ -โ”‚ โ€ข DQN prediction (confidence: 0.85) โ”‚ -โ”‚ โ€ข PPO prediction (confidence: 0.78) โ”‚ -โ”‚ โ€ข MAMBA-2 prediction (confidence: 0.92) โ”‚ -โ”‚ โ€ข TFT prediction (confidence: 0.81) โ”‚ -โ”‚ โ†’ Weighted average: ฮฃ(pred * conf) / ฮฃ(conf) โ”‚ -โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ - โ†“ -Trading Signal (Buy/Sell/Hold + position size) - โ†“ -โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” -โ”‚ Risk Validation โ”‚ -โ”‚ โ”‚ -โ”‚ โ€ข Kill switch check (circuit breaker) โ”‚ -โ”‚ โ€ข Position limit check (max 100K shares) โ”‚ -โ”‚ โ€ข Leverage check (max 4x) โ”‚ -โ”‚ โ€ข VaR check (portfolio risk) โ”‚ -โ”‚ โ€ข Confidence threshold (>0.7) โ”‚ -โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ - โ†“ -Order Submission (via TradingRepository) -``` - ---- - -## Integration Design: Enhanced ML Service - -### Current State (`services/trading_service/src/services/enhanced_ml.rs`) - -**Status**: โœ… **PRODUCTION READY** (Wave 160 Complete) - -```rust -pub struct EnhancedMLService { - inference_engine: Arc, - safety_manager: Arc, - model_performance_tracker: Arc>>, - ensemble_coordinator: Arc>, -} - -impl EnhancedMLService { - pub async fn get_trading_signal( - &self, - symbol: &Symbol, - market_data: &MarketData - ) -> Result { - // 1. Extract features - let features = self.extract_features(market_data)?; - - // 2. Get ensemble predictions (4 models) - let predictions = self.get_ensemble_predictions(&features).await?; - - // 3. Calculate confidence-weighted vote - let (ensemble_pred, ensemble_conf) = self.calculate_ensemble_vote(&predictions)?; - - // 4. Validate confidence threshold - if ensemble_conf < 0.7 { - return Err(MLError::LowConfidence { confidence: ensemble_conf }); - } - - // 5. Convert prediction to trading signal - let signal = self.prediction_to_signal(ensemble_pred, ensemble_conf, symbol)?; - - // 6. Validate signal safety - self.safety_manager.validate_signal(&signal).await?; - - Ok(signal) - } -} -``` - -**Key Implementation Details**: - -1. **Feature Extraction**: - ```rust - fn extract_features(&self, market_data: &MarketData) -> Result { - // Use UnifiedFinancialFeatures for 256-dim features - let features = UnifiedFinancialFeatures::extract_ml_features(market_data)?; - - // Validate feature dimensions - if features.len() != 256 { - return Err(MLError::FeatureDimensionMismatch { - expected: 256, - actual: features.len() - }); - } - - Ok(FeatureVector(features)) - } - ``` - -2. **Ensemble Predictions**: - ```rust - async fn get_ensemble_predictions( - &self, - features: &FeatureVector - ) -> Result> { - let mut predictions = Vec::with_capacity(4); - - // DQN prediction - if let Ok(pred) = self.inference_engine.predict("dqn_v1", features).await { - predictions.push(pred); - } - - // PPO prediction - if let Ok(pred) = self.inference_engine.predict("ppo_v1", features).await { - predictions.push(pred); - } - - // MAMBA-2 prediction - if let Ok(pred) = self.inference_engine.predict("mamba2_v1", features).await { - predictions.push(pred); - } - - // TFT prediction - if let Ok(pred) = self.inference_engine.predict("tft_v1", features).await { - predictions.push(pred); - } - - if predictions.is_empty() { - return Err(MLError::NoValidPredictions); - } - - Ok(predictions) - } - ``` - -3. **Ensemble Voting**: - ```rust - fn calculate_ensemble_vote( - &self, - predictions: &[RealPredictionResult] - ) -> Result<(Price, f64)> { - let total_confidence: f64 = predictions.iter() - .map(|p| p.confidence) - .sum(); - - if total_confidence == 0.0 { - return Err(MLError::ZeroConfidence); - } - - // Weighted average by confidence - let weighted_sum: f64 = predictions.iter() - .map(|p| p.prediction.to_f64() * p.confidence) - .sum(); - - let ensemble_prediction = weighted_sum / total_confidence; - let ensemble_confidence = predictions.iter() - .map(|p| p.confidence) - .sum::() / predictions.len() as f64; - - Ok((Price::from_f64(ensemble_prediction)?, ensemble_confidence)) - } - ``` - -4. **Signal Conversion**: - ```rust - fn prediction_to_signal( - &self, - prediction: Price, - confidence: f64, - symbol: &Symbol - ) -> Result { - // Current price from market data - let current_price = self.get_current_price(symbol)?; - - // Predicted return - let predicted_return = (prediction.to_f64() - current_price.to_f64()) - / current_price.to_f64(); - - // Signal direction - let side = if predicted_return > 0.01 { - OrderSide::Buy - } else if predicted_return < -0.01 { - OrderSide::Sell - } else { - return Ok(TradingSignal::Hold); - }; - - // Position sizing (confidence-based) - let base_quantity = 100.0; - let quantity = base_quantity * confidence; - - Ok(TradingSignal { - symbol: symbol.clone(), - side, - quantity: Decimal::from_f64(quantity)?, - strength: Decimal::from_f64(confidence)?, - reason: format!("ML ensemble prediction: {:.4}, confidence: {:.3}", - predicted_return, confidence), - }) - } - ``` - ---- - -## Error Handling Strategy - -### ML-Specific Errors - -```rust -#[derive(Error, Debug)] -pub enum MLIntegrationError { - #[error("ML inference failed: {reason}")] - InferenceFailed { reason: String }, - - #[error("Feature extraction failed: {reason}")] - FeatureExtractionFailed { reason: String }, - - #[error("Ensemble voting failed: no valid predictions")] - NoValidPredictions, - - #[error("Low confidence: {confidence:.3} < {threshold:.3}")] - LowConfidence { confidence: f64, threshold: f64 }, - - #[error("Model not loaded: {model_id}")] - ModelNotLoaded { model_id: String }, - - #[error("Model drift detected: {drift_score:.3} > {threshold:.3}")] - ModelDrift { drift_score: f64, threshold: f64 }, - - #[error("Safety validation failed: {reason}")] - SafetyViolation { reason: String }, -} -``` - -### Fallback Strategy - -``` -ML Inference Failure - โ†“ -โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” -โ”‚ Fallback Decision Tree โ”‚ -โ”‚ โ”‚ -โ”‚ 1. Cache hit? โ†’ Use cached prediction โ”‚ -โ”‚ 2. Partial ensemble? โ†’ Use available โ”‚ -โ”‚ models (โ‰ฅ2 required) โ”‚ -โ”‚ 3. All models failed? โ†’ Use rule- โ”‚ -โ”‚ based strategy (moving avg) โ”‚ -โ”‚ 4. Rule-based failed? โ†’ Hold position โ”‚ -โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ -``` - -**Implementation**: - -```rust -async fn get_trading_signal_with_fallback( - &self, - symbol: &Symbol, - market_data: &MarketData -) -> Result { - // Try ML inference - match self.get_trading_signal(symbol, market_data).await { - Ok(signal) => Ok(signal), - Err(e) => { - warn!("ML inference failed: {}, falling back to rule-based", e); - - // Fallback 1: Check cache - if let Some(cached_signal) = self.get_cached_signal(symbol).await { - info!("Using cached signal for {}", symbol); - return Ok(cached_signal); - } - - // Fallback 2: Rule-based strategy - self.get_rule_based_signal(symbol, market_data).await - } - } -} - -async fn get_rule_based_signal( - &self, - symbol: &Symbol, - market_data: &MarketData -) -> Result { - // Simple moving average crossover - let short_ma = self.calculate_ma(market_data, 5)?; - let long_ma = self.calculate_ma(market_data, 20)?; - - if short_ma > long_ma * 1.01 { - Ok(TradingSignal::buy(symbol.clone(), 100.0, 0.5)) - } else if short_ma < long_ma * 0.99 { - Ok(TradingSignal::sell(symbol.clone(), 100.0, 0.5)) - } else { - Ok(TradingSignal::Hold) - } -} -``` - ---- - -## Performance Monitoring - -### Metrics to Track - -```rust -pub struct MLPerformanceMetrics { - // Inference performance - pub inference_latency_p50: Duration, - pub inference_latency_p95: Duration, - pub inference_latency_p99: Duration, - - // Model accuracy - pub prediction_accuracy: f64, // % correct direction - pub sharpe_ratio: f64, // Risk-adjusted returns - pub win_rate: f64, // % profitable trades - pub avg_return_per_trade: f64, - - // Model health - pub drift_score: f64, // Model drift detection - pub avg_confidence: f64, // Average prediction confidence - pub cache_hit_rate: f64, // Prediction cache efficiency - - // System health - pub gpu_utilization: f64, // GPU usage % - pub gpu_memory_used: usize, // GPU VRAM in bytes - pub failed_predictions: u64, // Error count - pub fallback_invocations: u64, // Rule-based fallbacks -} -``` - -### Prometheus Integration - -```rust -lazy_static! { - static ref ML_SIGNAL_LATENCY: Histogram = register_histogram!( - "foxhunt_ml_signal_latency_microseconds", - "ML trading signal generation latency" - ).unwrap(); - - static ref ML_SIGNAL_ACCURACY: Gauge = register_gauge!( - "foxhunt_ml_signal_accuracy", - "ML trading signal accuracy (rolling 100 trades)" - ).unwrap(); - - static ref ML_FALLBACK_COUNTER: Counter = register_counter!( - "foxhunt_ml_fallback_total", - "Total ML fallbacks to rule-based strategy" - ).unwrap(); - - static ref ML_ENSEMBLE_CONFIDENCE: Gauge = register_gauge!( - "foxhunt_ml_ensemble_confidence", - "Average ensemble prediction confidence" - ).unwrap(); -} -``` - ---- - -## Implementation Plan (Agents 10.10-10.13) - -### Agent 10.10: TDD Test Suite (RED Phase) - -**Objective**: Write comprehensive failing tests defining ML integration behavior - -**Test Categories**: - -1. **Feature Extraction Tests** (`tests/ml_integration/feature_extraction_tests.rs`): - ```rust - #[tokio::test] - async fn test_feature_extraction_256_dimensions() { - // Should extract exactly 256 features from market data - } - - #[tokio::test] - async fn test_feature_extraction_handles_missing_data() { - // Should handle missing OHLCV data gracefully - } - - #[tokio::test] - async fn test_feature_validation_rejects_nan() { - // Should reject features with NaN/Inf values - } - ``` - -2. **Ensemble Prediction Tests** (`tests/ml_integration/ensemble_tests.rs`): - ```rust - #[tokio::test] - async fn test_ensemble_voting_confidence_weighted() { - // Should weight predictions by confidence scores - } - - #[tokio::test] - async fn test_ensemble_requires_minimum_models() { - // Should require โ‰ฅ2 models for ensemble vote - } - - #[tokio::test] - async fn test_ensemble_rejects_low_confidence() { - // Should reject predictions with confidence <0.7 - } - ``` - -3. **Signal Conversion Tests** (`tests/ml_integration/signal_conversion_tests.rs`): - ```rust - #[tokio::test] - async fn test_prediction_to_buy_signal() { - // Should convert bullish prediction to Buy signal - } - - #[tokio::test] - async fn test_prediction_to_sell_signal() { - // Should convert bearish prediction to Sell signal - } - - #[tokio::test] - async fn test_confidence_based_position_sizing() { - // Should scale position size with confidence - } - ``` - -4. **Fallback Strategy Tests** (`tests/ml_integration/fallback_tests.rs`): - ```rust - #[tokio::test] - async fn test_fallback_to_cache_on_inference_failure() { - // Should use cached signal when inference fails - } - - #[tokio::test] - async fn test_fallback_to_rule_based_on_all_models_failed() { - // Should use moving average when all ML models fail - } - - #[tokio::test] - async fn test_fallback_to_hold_on_complete_failure() { - // Should hold position when all strategies fail - } - ``` - -5. **Integration Tests** (`tests/ml_integration/end_to_end_tests.rs`): - ```rust - #[tokio::test] - async fn test_ml_strategy_full_pipeline() { - // Market data โ†’ Features โ†’ Predictions โ†’ Signal โ†’ Order - } - - #[tokio::test] - async fn test_ml_strategy_with_kill_switch() { - // Should respect kill switch during ML trading - } - - #[tokio::test] - async fn test_ml_strategy_concurrent_predictions() { - // Should handle concurrent predictions for multiple symbols - } - ``` - -**Deliverable**: 30+ failing tests defining ML integration contract - ---- - -### Agent 10.11: Core ML Integration (GREEN Phase) - -**Objective**: Implement minimal code to pass Agent 10.10 tests - -**Files to Modify**: - -1. **`services/trading_service/src/services/enhanced_ml.rs`**: - - Implement `extract_features()` using `UnifiedFinancialFeatures` - - Implement `get_ensemble_predictions()` calling `RealMLInferenceEngine` - - Implement `calculate_ensemble_vote()` with confidence weighting - - Implement `prediction_to_signal()` with position sizing - -2. **`services/trading_service/src/ml_strategy_executor.rs`** (NEW): - ```rust - pub struct MLStrategyExecutor { - enhanced_ml_service: Arc, - fallback_strategy: Arc, - performance_tracker: Arc>, - } - - impl MLStrategyExecutor { - pub async fn execute( - &self, - symbol: &Symbol, - market_data: &MarketData - ) -> Result; - } - ``` - -3. **`services/trading_service/src/services/trading.rs`**: - - Modify `submit_order()` to accept ML-generated signals - - Add ML performance metrics logging - - Integrate with kill switch validation - -**Success Criteria**: All Agent 10.10 tests pass (GREEN) - ---- - -### Agent 10.12: Production Hardening (REFACTOR Phase) - -**Objective**: Improve code quality, add error handling, optimize performance - -**Enhancements**: - -1. **Error Handling**: - - Add structured error types (`MLIntegrationError`) - - Implement graceful degradation (fallback chain) - - Add retry logic for transient failures (network, GPU) - -2. **Performance Optimization**: - - Add prediction caching (60-second TTL) - - Batch feature extraction for multiple symbols - - Optimize ensemble voting (parallel predictions) - -3. **Monitoring**: - - Add Prometheus metrics export - - Implement performance tracking (latency, accuracy) - - Add drift detection alerts - -4. **Documentation**: - - Document ML integration architecture - - Add code examples for strategy development - - Create troubleshooting guide - -**Success Criteria**: -- All tests still pass (GREEN maintained) -- Code coverage >80% -- No performance regressions - ---- - -### Agent 10.13: End-to-End Validation - -**Objective**: Validate ML integration with production scenarios - -**Validation Tests**: - -1. **Backtest Validation** (`tests/e2e/ml_backtest_validation.rs`): - ```rust - #[tokio::test] - async fn test_ml_strategy_backtest_es_fut() { - // Backtest ML strategy on ES.FUT historical data - // Expected: Sharpe >1.0, win rate >55% - } - - #[tokio::test] - async fn test_ml_strategy_vs_rule_based() { - // Compare ML vs moving average on same data - // Expected: ML outperforms by โ‰ฅ10% returns - } - ``` - -2. **Stress Testing** (`tests/e2e/ml_stress_tests.rs`): - ```rust - #[tokio::test] - async fn test_ml_strategy_high_frequency() { - // 1000 predictions/second for 1 minute - // Expected: P99 latency <100ฮผs - } - - #[tokio::test] - async fn test_ml_strategy_model_failure() { - // Simulate GPU failure mid-trading - // Expected: Fallback to CPU, no orders lost - } - ``` - -3. **Compliance Testing** (`tests/e2e/ml_compliance_tests.rs`): - ```rust - #[tokio::test] - async fn test_ml_strategy_kill_switch_integration() { - // Verify kill switch halts ML trading - } - - #[tokio::test] - async fn test_ml_strategy_audit_logging() { - // Verify all ML predictions are logged - } - ``` - -**Success Criteria**: -- All E2E tests pass -- Production-ready deployment checklist complete -- Documentation updated with ML strategy guide - ---- - -## Deployment Checklist - -### Pre-Deployment - -- [ ] All Agent 10.10-10.13 tests pass (100%) -- [ ] Code coverage >80% for ML integration -- [ ] Benchmark ML strategy vs rule-based (>10% improvement) -- [ ] GPU training complete (DQN, PPO, MAMBA-2, TFT) -- [ ] Models uploaded to MinIO checkpoint storage -- [ ] Prometheus dashboards configured -- [ ] Alert rules configured (drift, latency, accuracy) - -### Deployment - -- [ ] Deploy ML models to production GPU server -- [ ] Load models into `RealMLInferenceEngine` -- [ ] Enable ML strategy in trading service config -- [ ] Monitor performance for 24 hours (paper trading) -- [ ] Validate metrics (latency, accuracy, Sharpe) -- [ ] Enable live trading with 10% allocation - -### Post-Deployment - -- [ ] Monitor Prometheus dashboards daily -- [ ] Review ML performance metrics weekly -- [ ] Retrain models monthly (90-day window) -- [ ] Audit compliance logging quarterly - ---- - -## Risk Mitigation - -### ML-Specific Risks - -| Risk | Impact | Mitigation | -|------|--------|-----------| -| Model overfitting | High | Use 70/20/10 train/val/test split, early stopping | -| Drift detection | High | Monitor drift score <0.1, retrain monthly | -| GPU failure | Medium | CPU fallback, rule-based fallback | -| Low confidence | Medium | Reject signals with confidence <0.7 | -| Inference timeout | Low | 50ฮผs timeout, cache previous predictions | -| Feature extraction failure | Low | Validate 256-dim features, handle missing data | - -### Trading Risks - -| Risk | Impact | Mitigation | -|------|--------|-----------| -| Kill switch bypass | Critical | First validation in `submit_order()` | -| Position limit violation | High | Validate against RiskManager before order | -| Leverage limit violation | High | Check max 4x leverage | -| VaR limit violation | Medium | Calculate portfolio VaR after each trade | -| Overtrading | Medium | Rate limit ML signals (max 10/min per symbol) | - ---- - -## Performance Expectations - -### Latency Targets - -| Operation | Target | P95 | P99 | -|-----------|--------|-----|-----| -| Feature extraction | <5ฮผs | 10ฮผs | 20ฮผs | -| ML inference (single model) | <50ฮผs | 75ฮผs | 100ฮผs | -| Ensemble voting (4 models) | <200ฮผs | 300ฮผs | 500ฮผs | -| Signal conversion | <10ฮผs | 20ฮผs | 30ฮผs | -| **End-to-end signal generation** | **<250ฮผs** | **400ฮผs** | **600ฮผs** | - -### Accuracy Targets - -| Metric | Target | Baseline (Rule-Based) | -|--------|--------|----------------------| -| Prediction accuracy | >60% | 52% | -| Sharpe ratio | >1.5 | 0.8 | -| Win rate | >55% | 48% | -| Max drawdown | <15% | 22% | -| Returns (annualized) | >25% | 12% | - ---- - -## Code Examples - -### Example 1: ML Strategy in Backtest - -```rust -use trading_service::ml_strategy_executor::MLStrategyExecutor; -use ml::inference::{RealMLInferenceEngine, RealInferenceConfig}; - -#[tokio::main] -async fn main() -> Result<()> { - // Initialize ML inference engine - let config = RealInferenceConfig::default(); - let safety_manager = Arc::new(MLSafetyManager::new(MLSafetyConfig::default())); - let engine = Arc::new(RealMLInferenceEngine::new(config, safety_manager)); - - // Load trained models - engine.load_model("dqn_v1".to_string(), dqn_config).await?; - engine.load_model("ppo_v1".to_string(), ppo_config).await?; - engine.load_model("mamba2_v1".to_string(), mamba2_config).await?; - engine.load_model("tft_v1".to_string(), tft_config).await?; - - // Create ML strategy executor - let enhanced_ml = Arc::new(EnhancedMLService::new(engine)); - let executor = MLStrategyExecutor::new(enhanced_ml); - - // Execute strategy on historical data - let market_data = load_market_data("ES.FUT", start_date, end_date)?; - - for bar in market_data { - let signal = executor.execute(&bar.symbol, &bar).await?; - - match signal { - TradingSignal::Buy { quantity, strength, .. } => { - println!("BUY {} @ confidence {:.3}", quantity, strength); - } - TradingSignal::Sell { quantity, strength, .. } => { - println!("SELL {} @ confidence {:.3}", quantity, strength); - } - TradingSignal::Hold => { - println!("HOLD"); - } - } - } - - Ok(()) -} -``` - -### Example 2: ML Strategy in Paper Trading - -```rust -use trading_service::services::trading::TradingServiceImpl; -use trading_service::ml_strategy_executor::MLStrategyExecutor; - -#[tokio::main] -async fn main() -> Result<()> { - // Initialize trading service with ML strategy - let state = TradingServiceState::new(config).await?; - let trading_service = TradingServiceImpl::new(Arc::new(state)); - - // Load ML models - let ml_executor = MLStrategyExecutor::load_from_checkpoint("checkpoints/latest")?; - - // Paper trading loop - loop { - // Get real-time market data - let market_data = get_real_time_data("ES.FUT").await?; - - // Get ML trading signal - let signal = ml_executor.execute(&Symbol::from("ES.FUT"), &market_data).await?; - - // Submit order if signal is actionable - if let TradingSignal::Buy { quantity, .. } | TradingSignal::Sell { quantity, .. } = signal { - let request = SubmitOrderRequest { - account_id: "paper_trading".to_string(), - symbol: "ES.FUT".to_string(), - side: signal.side as i32, - quantity: quantity.to_f64(), - order_type: OrderType::Market as i32, - price: None, - stop_price: None, - }; - - let response = trading_service.submit_order(Request::new(request)).await?; - println!("Order submitted: {:?}", response); - } - - // Sleep until next bar - tokio::time::sleep(Duration::from_secs(60)).await; - } -} -``` - ---- - -## Appendix: ML Model Training Status - -### Model Readiness (Wave 160 Complete) - -| Model | Status | Training Data | Performance | Latency | Memory | -|-------|--------|---------------|-------------|---------|--------| -| **MAMBA-2** | โœ… READY | 200 epochs, ES.FUT | 70.6% loss reduction | 0.56s/epoch | <1GB | -| **DQN** | โณ READY (needs training) | - | TBD | <50ฮผs (target) | 50-150MB | -| **PPO** | โณ READY (needs training) | - | TBD | <50ฮผs (target) | 50-200MB | -| **TFT** | โณ READY (needs training) | - | TBD | <100ฮผs (target) | 1.5-2.5GB | - -### Training Timeline (Post-Wave 160) - -1. **Week 1-2**: DQN training (ES.FUT, NQ.FUT, 90 days) -2. **Week 2-3**: PPO training (ES.FUT, NQ.FUT, 90 days) -3. **Week 3-5**: TFT training (ES.FUT, NQ.FUT, 90 days) -4. **Week 5-6**: Ensemble validation, hyperparameter tuning - -**Total Timeline**: 6 weeks for production-ready ensemble - ---- - -## Conclusion - -This design provides a comprehensive roadmap for integrating the ML inference engine with the trading service. The TDD methodology ensures robust, testable code with clear acceptance criteria at each phase. - -**Next Steps**: -1. Agent 10.10: Implement failing test suite (RED) -2. Agent 10.11: Implement core integration (GREEN) -3. Agent 10.12: Production hardening (REFACTOR) -4. Agent 10.13: End-to-end validation - -**Key Success Metrics**: -- โœ… All tests pass (100% coverage) -- โœ… Latency <250ฮผs end-to-end -- โœ… Sharpe ratio >1.5 -- โœ… GPU memory <1GB -- โœ… Production deployment ready - ---- - -**Document Version**: 1.0 -**Last Updated**: 2025-10-15 -**Authors**: Agent 10.9 (Claude Code) -**Review Status**: Ready for Wave 10 Agents 10.10-10.13 diff --git a/services/trading_service/src/streaming/README.md b/services/trading_service/src/streaming/README.md deleted file mode 100644 index b78f2092e..000000000 --- a/services/trading_service/src/streaming/README.md +++ /dev/null @@ -1,291 +0,0 @@ -# gRPC Streaming Backpressure Monitoring - -## Overview - -Wave 67 Agent 6 implementation: Comprehensive backpressure monitoring and handling for all gRPC streaming methods in the trading service. - -## Components - -### 1. BackpressureMonitor (`backpressure.rs`) - -Provides real-time buffer utilization monitoring with configurable thresholds: - -- **Warning threshold**: 80% utilization (default) -- **Critical threshold**: 95% utilization (default) -- **Full buffer**: 100% utilization - -**Features**: -- Lock-free atomic counters for minimal overhead (<100ns per check) -- Configurable thresholds per stream type -- Automatic logging at warning/critical levels -- Status tracking (Healthy, Warning, Critical, Full) - -**Usage**: -```rust -let config = BackpressureConfig { - buffer_capacity: 1000, - warning_threshold: 0.8, - critical_threshold: 0.95, - metric_prefix: "orders".to_string(), -}; -let monitor = BackpressureMonitor::new(config); - -// Check buffer status -let status = monitor.check(current_buffer_size); -if status.is_warning() { - monitor.log_status(&status, "orders"); -} -``` - -### 2. StreamMetrics (`metrics.rs`) - -Prometheus metrics integration for stream observability: - -**Metrics Exposed**: -- `stream_buffer_utilization_percent{stream_name}` - Current buffer utilization -- `stream_messages_sent_total{stream_name}` - Total messages sent -- `stream_messages_dropped_total{stream_name, reason}` - Dropped messages by reason -- `stream_send_timeouts_total{stream_name}` - Send timeout events -- `stream_backpressure_warnings_total{stream_name}` - Warning events -- `stream_backpressure_critical_total{stream_name}` - Critical events - -**Usage**: -```rust -let metrics = StreamMetrics::new("orders"); -metrics.set_buffer_utilization(75.0); -metrics.inc_messages_sent(); -metrics.inc_backpressure_warnings(); -``` - -### 3. MonitoredSender (`monitored_channel.rs`) - -Instrumented channel wrapper with timeout-based sends: - -**Features**: -- Automatic backpressure monitoring on every send -- Configurable send timeouts (default: 100ms) -- Graceful degradation with logging -- Best-effort delivery mode for high-frequency data -- Drop-in replacement for `mpsc::Sender` - -**Usage**: -```rust -// Create monitored channel -let (tx, rx, monitor, metrics) = create_monitored_channel(1000, "orders"); - -// Send with timeout and backpressure handling -match tx.send_monitored(order_event).await { - Ok(()) => {}, // Success - Err(e) => warn!("Send failed: {}", e), -} - -// Best-effort send (doesn't fail on drop) -tx.send_best_effort(market_data_event).await; -``` - -## Integration Pattern - -### Trading Service Streaming Methods - -All 12 streaming methods now use monitored channels: - -1. **Orders Stream** (`stream_orders`): - - Buffer: Medium frequency (10K) - - Mode: Monitored send with warnings - - Backpressure: Fail-fast on timeout - -2. **Positions Stream** (`stream_positions`): - - Buffer: Medium frequency (10K) - - Mode: Best-effort delivery - - Backpressure: Silent drops with metrics - -3. **Market Data Stream** (`stream_market_data`): - - Buffer: High frequency (100K) - - Mode: Best-effort delivery - - Backpressure: Expected under high load - -4. **Executions Stream** (`stream_executions`): - - Buffer: Low frequency (1K) - - Mode: Monitored send - - Backpressure: Critical alerts - -### Example Integration - -```rust -async fn stream_orders( - &self, - request: Request, -) -> TonicResult> { - let req = request.into_inner(); - - // Create monitored channel - use crate::streaming::StreamType; - let buffer_size = StreamType::MediumFrequency.buffer_size(); - let (tx, rx, _monitor, _metrics) = create_monitored_channel(buffer_size, "orders"); - - // Spawn subscription task - let event_publisher = Arc::clone(&self.state.event_publisher); - tokio::spawn(async move { - let mut subscription = event_publisher.subscribe_orders(req.account_id).await; - while let Some(order_event) = subscription.next().await { - // Send with backpressure handling - if let Err(e) = tx.send_monitored(order_event).await { - warn!("Order stream send failed: {}", e); - break; - } - } - }); - - let stream = ReceiverStream::new(rx); - Ok(Response::new(Box::pin(stream))) -} -``` - -## Performance Characteristics - -### Overhead Measurements - -- **Backpressure check**: <100ns per operation -- **Metrics update**: <50ns (atomic operations only) -- **Total overhead**: <150ns per message (well within 14ns latency budget for HFT) - -### Buffer Utilization Thresholds - -| Threshold | Utilization | Action | -|-----------|-------------|--------| -| Healthy | 0-79% | Normal operation | -| Warning | 80-94% | Log warning, inc metrics | -| Critical | 95-99% | Log error, inc metrics | -| Full | 100% | Drop message, return error | - -### Send Timeout Strategy - -| Stream Type | Timeout | Rationale | -|-------------|---------|-----------| -| Low frequency | 100ms | Balance responsiveness/reliability | -| Medium frequency | 100ms | Standard timeout for order/execution streams | -| High frequency | 50ms | Faster timeout for market data | - -## Observability - -### Prometheus Alerts - -Recommended alerting rules: - -```yaml -groups: - - name: streaming_backpressure - rules: - # Alert on sustained high utilization - - alert: StreamBufferHighUtilization - expr: stream_buffer_utilization_percent > 80 - for: 1m - annotations: - summary: "Stream {{ $labels.stream_name }} buffer at {{ $value }}%" - - # Alert on message drops - - alert: StreamMessagesDropped - expr: rate(stream_messages_dropped_total[1m]) > 0 - annotations: - summary: "Stream {{ $labels.stream_name }} dropping messages" - - # Alert on frequent timeouts - - alert: StreamSendTimeouts - expr: rate(stream_send_timeouts_total[1m]) > 10 - for: 5m - annotations: - summary: "Stream {{ $labels.stream_name }} experiencing frequent timeouts" -``` - -### Grafana Dashboards - -Key metrics to monitor: -1. Buffer utilization by stream (gauge) -2. Messages sent/dropped rate (counter) -3. Send timeout rate (counter) -4. Backpressure events (warnings + critical) - -## Testing - -### Unit Tests - -All modules include comprehensive unit tests: -- Backpressure status transitions -- Message counting accuracy -- Timeout behavior -- Best-effort delivery mode - -Run tests: -```bash -cargo test --package trading_service --lib streaming -``` - -### Load Testing - -Simulate backpressure scenarios: -```rust -#[tokio::test] -async fn test_backpressure_under_load() { - let (tx, _rx, monitor, metrics) = create_monitored_channel(10, "test"); - - // Fill buffer - for i in 0..10 { - tx.send_monitored(i).await.expect("Should succeed"); - } - - // This should timeout - let result = tx.send_monitored(11).await; - assert!(result.is_err()); - assert_eq!(monitor.messages_dropped(), 1); -} -``` - -## Migration Guide - -### Before (Silent Failures) -```rust -let (tx, rx) = mpsc::channel(1000); - -tokio::spawn(async move { - while let Some(event) = subscription.next().await { - let _ = tx.send(event).await; // Silent failure! - } -}); -``` - -### After (Observable Backpressure) -```rust -let (tx, rx, _monitor, _metrics) = create_monitored_channel(1000, "stream_name"); - -tokio::spawn(async move { - while let Some(event) = subscription.next().await { - if let Err(e) = tx.send_monitored(event).await { - warn!("Send failed: {}", e); // Observable failure - break; - } - } -}); -``` - -## Future Enhancements - -1. **Adaptive Buffer Sizing**: Dynamically adjust buffer sizes based on historical utilization -2. **Client-Side Backpressure**: Propagate backpressure signals to gRPC clients -3. **Priority Queues**: Multiple priority levels within streams -4. **Circuit Breaker**: Automatic stream suspension on persistent failures -5. **Enhanced Metrics**: P95/P99 latency percentiles for send operations - -## References - -- Wave 66 Agent 9: Identified missing backpressure handling -- Wave 67 Agent 3: HTTP/2 streaming performance optimizations -- Wave 67 Agent 6: This implementation - -## Architecture Compliance - -Follows CLAUDE.md principles: -- โœ… Central configuration management (StreamType enum) -- โœ… Service architecture (trading service uses streaming module) -- โœ… Prometheus metrics integration -- โœ… Production-ready observability -- โœ… Performance targets maintained (<150ns overhead) diff --git a/services/trading_service/tests/common/README.md b/services/trading_service/tests/common/README.md deleted file mode 100644 index a61d8f8c8..000000000 --- a/services/trading_service/tests/common/README.md +++ /dev/null @@ -1,372 +0,0 @@ -# JWT Authentication Helpers for E2E Tests - -This module provides reusable JWT authentication utilities for integration and end-to-end tests across the Foxhunt trading system. - -## Overview - -The `auth_helpers` module centralizes JWT token generation and authenticated client creation to ensure consistency across test suites. It provides: - -- โœ… Valid JWT token generation matching API Gateway validation -- โœ… Authentication interceptor for gRPC clients -- โœ… Support for both MFA-enabled and MFA-disabled scenarios -- โœ… Configurable user roles and permissions -- โœ… Environment-based JWT secret management - -## Quick Start - -### Basic Usage - -```rust -use common::auth_helpers::{create_default_test_jwt, create_auth_interceptor, TestAuthConfig}; -use tonic::transport::Channel; - -#[tokio::test] -async fn test_authenticated_request() -> Result<()> { - // 1. Create JWT token - let token = create_default_test_jwt()?; - - // 2. Create interceptor with authentication - let interceptor = create_auth_interceptor(TestAuthConfig::trader())?; - - // 3. Connect to API Gateway - let channel = Channel::from_static("http://localhost:50051") - .connect() - .await?; - - // 4. Create authenticated client - let mut client = TradingServiceClient::with_interceptor(channel, interceptor); - - // 5. Make authenticated request - let response = client.get_account_info(Request::new(GetAccountInfoRequest {})).await?; - - Ok(()) -} -``` - -## Core Functions - -### `create_default_test_jwt() -> Result` - -Generates a JWT token with default trader configuration. - -**Example:** -```rust -let token = create_default_test_jwt()?; -// Use token in Authorization header: Bearer -``` - -### `create_test_jwt(config: TestAuthConfig) -> Result` - -Generates a JWT token with custom configuration. - -**Example:** -```rust -let config = TestAuthConfig::admin() - .with_user_id("admin_user_123") - .with_expiry(Duration::minutes(30)); - -let token = create_test_jwt(config)?; -``` - -### `create_auth_interceptor(config: TestAuthConfig) -> Result` - -Creates an authentication interceptor that injects JWT token and user context into all requests. - -**Example:** -```rust -let interceptor = create_auth_interceptor(TestAuthConfig::trader())?; -let channel = Channel::from_static("http://localhost:50051").connect().await?; -let mut client = TradingServiceClient::with_interceptor(channel, interceptor); -``` - -## Configuration Options - -### TestAuthConfig Presets - -#### Trader (Default) -```rust -let config = TestAuthConfig::trader(); -// - user_id: "test_trader_001" -// - roles: ["trader"] -// - permissions: ["api.access", "trading.submit", "trading.view", "trading.cancel"] -// - MFA: disabled -``` - -#### Admin -```rust -let config = TestAuthConfig::admin(); -// - user_id: "test_admin_001" -// - roles: ["admin"] -// - permissions: ["api.access", "trading.*", "admin.*"] -// - MFA: enabled and verified -``` - -#### Viewer (Read-only) -```rust -let config = TestAuthConfig::viewer(); -// - user_id: "test_viewer_001" -// - roles: ["viewer"] -// - permissions: ["api.access", "trading.view"] -// - MFA: disabled -``` - -### Builder Pattern - -Customize any configuration: - -```rust -let config = TestAuthConfig::trader() - .with_user_id("custom_user_123") - .with_roles(vec!["custom_role".to_string()]) - .with_permissions(vec!["custom.permission".to_string()]) - .with_expiry(Duration::minutes(30)) - .with_mfa_enabled(); -``` - -## Advanced Usage - -### Testing MFA Scenarios - -#### MFA Enabled and Verified -```rust -let config = TestAuthConfig::trader().with_mfa_enabled(); -let token = create_test_jwt(config)?; -// Token indicates MFA is verified -``` - -#### MFA Enabled but Not Verified (for testing MFA flows) -```rust -let config = TestAuthConfig::trader().with_mfa_unverified(); -let token = create_test_jwt(config)?; -// Token indicates MFA needs verification -``` - -### Testing Token Validation - -#### Expired Token -```rust -let expired_token = create_expired_test_jwt()?; -// This token will fail API Gateway validation due to expiry -``` - -#### Invalid Issuer -```rust -let invalid_token = create_invalid_issuer_jwt()?; -// This token will fail API Gateway validation due to wrong issuer -``` - -### Multiple Authenticated Clients - -```rust -#[tokio::test] -async fn test_multiple_users() -> Result<()> { - // Create trader client - let trader_interceptor = create_auth_interceptor(TestAuthConfig::trader())?; - let channel1 = Channel::from_static("http://localhost:50051").connect().await?; - let mut trader_client = TradingServiceClient::with_interceptor(channel1, trader_interceptor); - - // Create admin client - let admin_interceptor = create_auth_interceptor(TestAuthConfig::admin())?; - let channel2 = Channel::from_static("http://localhost:50051").connect().await?; - let mut admin_client = TradingServiceClient::with_interceptor(channel2, admin_interceptor); - - // Use both clients in test - trader_client.submit_order(...).await?; - admin_client.cancel_all_orders(...).await?; - - Ok(()) -} -``` - -## Environment Variables - -The auth helpers respect the following environment variables: - -### `JWT_SECRET` -Override the default test JWT secret. Useful for CI/CD environments. - -**Default:** `m5G2uUIX1DzSYYny8hXkF93udN5s3Tq5oFWSrGtCqyTaUeYwslf/9sh6UxF5KM5AF0PwlbRWmXHAvbCF73V+Dw==` - -```bash -export JWT_SECRET="your-custom-secret-for-testing" -cargo test -``` - -### `API_GATEWAY_ADDR` -Override the default API Gateway address. - -**Default:** `http://localhost:50051` - -```bash -export API_GATEWAY_ADDR="http://api-gateway:50051" -cargo test -``` - -## Helper Functions - -### `get_test_user_id() -> String` -Returns the default test user ID: `"test_trader_001"` - -### `get_test_jwt_secret() -> String` -Returns the JWT secret from environment or default test secret. - -### `get_api_gateway_addr() -> String` -Returns the API Gateway address from environment or default. - -## JWT Token Structure - -Generated tokens match API Gateway validation requirements: - -```rust -{ - "jti": "unique-jwt-id", // Required for revocation tracking - "sub": "test_trader_001", // User ID - "iat": 1234567890, // Issued at (Unix timestamp) - "exp": 1234571490, // Expiry (Unix timestamp) - "iss": "foxhunt-trading", // Issuer (must match API Gateway) - "aud": "trading-api", // Audience (must match API Gateway) - "roles": ["trader"], // User roles - "permissions": [ // User permissions - "api.access", - "trading.submit", - "trading.view" - ], - "token_type": "access", // Token type - "session_id": "session-uuid" // Session ID -} -``` - -## Integration with Existing Tests - -### Example: Trading Service E2E Test - -```rust -mod common; - -use common::auth_helpers::{create_auth_interceptor, TestAuthConfig}; -use tonic::transport::Channel; - -#[tokio::test] -async fn test_order_lifecycle() -> Result<()> { - // Setup authenticated client - let interceptor = create_auth_interceptor(TestAuthConfig::trader())?; - let channel = Channel::from_static("http://localhost:50051") - .connect() - .await?; - let mut client = TradingServiceClient::with_interceptor(channel, interceptor); - - // Submit order - let order_request = SubmitOrderRequest { - symbol: "AAPL".to_string(), - side: OrderSide::Buy as i32, - order_type: OrderType::Limit as i32, - quantity: 100, - price: Some(150.0), - }; - - let response = client.submit_order(Request::new(order_request)).await?; - let order_id = response.into_inner().order_id; - - // Check order status - let status_request = GetOrderStatusRequest { order_id }; - let status = client.get_order_status(Request::new(status_request)).await?; - - assert_eq!(status.into_inner().status, OrderStatus::Pending as i32); - - Ok(()) -} -``` - -### Example: Testing Authorization Failures - -```rust -#[tokio::test] -async fn test_insufficient_permissions() -> Result<()> { - // Create viewer client (read-only permissions) - let interceptor = create_auth_interceptor(TestAuthConfig::viewer())?; - let channel = Channel::from_static("http://localhost:50051").connect().await?; - let mut client = TradingServiceClient::with_interceptor(channel, interceptor); - - // Attempt to submit order (should fail) - let order_request = SubmitOrderRequest { /* ... */ }; - let result = client.submit_order(Request::new(order_request)).await; - - // Verify authorization failure - assert!(result.is_err()); - assert!(matches!( - result.unwrap_err().code(), - tonic::Code::PermissionDenied - )); - - Ok(()) -} -``` - -## Testing Checklist - -When writing E2E tests with authentication: - -- [ ] Use `create_auth_interceptor()` for authenticated clients -- [ ] Choose appropriate `TestAuthConfig` preset (trader/admin/viewer) -- [ ] Customize config with builder pattern if needed -- [ ] Test both successful and failed authorization scenarios -- [ ] Consider MFA scenarios if relevant -- [ ] Use environment variables for CI/CD flexibility -- [ ] Verify JWT token structure matches API Gateway requirements - -## Troubleshooting - -### "Invalid JWT token" errors - -**Cause:** Token doesn't match API Gateway validation requirements. - -**Solution:** Ensure: -- Issuer is `"foxhunt-trading"` -- Audience is `"trading-api"` -- Token is not expired -- JWT secret matches API Gateway configuration - -### "Failed to connect to API Gateway" errors - -**Cause:** API Gateway is not running or address is incorrect. - -**Solution:** -```bash -# Check API Gateway is running -docker-compose ps api_gateway - -# Verify address is correct -echo $API_GATEWAY_ADDR # Should be http://localhost:50051 - -# Restart API Gateway if needed -docker-compose restart api_gateway -``` - -### "Permission denied" errors - -**Cause:** User lacks required permissions for the operation. - -**Solution:** Use appropriate config: -```rust -// For admin operations -let config = TestAuthConfig::admin(); - -// For custom permissions -let config = TestAuthConfig::trader() - .with_permissions(vec!["admin.cancel_all".to_string()]); -``` - -## Best Practices - -1. **Reuse configurations:** Use presets (trader/admin/viewer) when possible -2. **Explicit permissions:** When testing authorization, explicitly set permissions to verify access control -3. **Test both success and failure:** Test with correct and insufficient permissions -4. **Environment isolation:** Use environment variables for CI/CD to avoid hardcoded credentials -5. **Token uniqueness:** Each token has unique JTI for revocation tracking -6. **MFA testing:** Test both MFA-enabled and MFA-disabled flows where applicable - -## See Also - -- [CLAUDE.md](/home/jgrusewski/Work/foxhunt/CLAUDE.md) - System architecture and infrastructure -- [API Gateway Authentication](/home/jgrusewski/Work/foxhunt/services/api_gateway/src/auth/jwt/service.rs) - JWT validation implementation -- [MFA Module](/home/jgrusewski/Work/foxhunt/services/api_gateway/src/auth/mfa/mod.rs) - Multi-factor authentication diff --git a/testing/api-gateway-load/CHEAT_SHEET.md b/testing/api-gateway-load/CHEAT_SHEET.md deleted file mode 100644 index 689491fbf..000000000 --- a/testing/api-gateway-load/CHEAT_SHEET.md +++ /dev/null @@ -1,161 +0,0 @@ -# Load Testing Cheat Sheet - -## Prerequisites Check -```bash -# Docker services running? -docker ps | grep -E "(redis|postgres)" | wc -l # Should be 2+ - -# API Gateway running? -curl -f http://localhost:50050/health && echo "โœ… Ready" || echo "โŒ Not running" - -# Load tests built? -ls -lah ../../target/release/load_test_runner && echo "โœ… Built" || echo "โŒ Not built" -``` - -## Quick Commands - -### Run Tests -```bash -# Normal Load (1K clients, 60s) -cargo run -r --bin load_test_runner -- normal - -# Spike Load (0โ†’10K clients) -cargo run -r --bin load_test_runner -- spike - -# Stress Test (find breaking point) -cargo run -r --bin load_test_runner -- stress - -# All tests -cargo run -r --bin load_test_runner -- all -``` - -### View Results -```bash -firefox *.html # Open all reports -``` - -### Extract Metrics -```bash -# P99 Latency -grep -oP 'P99 Latency.*?value">\K[^<]+' normal_load_report.html - -# Throughput -grep -oP 'Requests/Second.*?value">\K[^<]+' normal_load_report.html - -# Error Rate -grep -oP 'Error Rate.*?value">\K[^<]+' normal_load_report.html -``` - -## Performance Targets - -| Metric | Target | Pass/Fail | -|--------|--------|-----------| -| P99 Latency | <10ฮผs | โœ… / โŒ | -| Throughput | >100K req/s | โœ… / โŒ | -| Error Rate | <0.1% | โœ… / โŒ | - -## Troubleshooting - -### Connection Refused -```bash -# Start API Gateway -cargo run -r -p api_gateway -``` - -### High Latency -```bash -# Check CPU -top -p $(pgrep api_gateway) - -# Check DB connections -psql -h localhost -p 5433 -U postgres -c 'SELECT count(*) FROM pg_stat_activity;' -``` - -### High Errors -```bash -# Check logs -docker logs foxhunt-api-gateway --tail 50 - -# Check backend health -curl http://localhost:50051/health # Trading -curl http://localhost:50052/health # Backtesting -curl http://localhost:50053/health # ML Training -``` - -## 5-Minute Validation - -```bash -# 1. Start services -docker-compose up -d redis postgres - -# 2. Start API Gateway -cargo run -r -p api_gateway & - -# 3. Wait for startup -sleep 5 - -# 4. Run quick test -cd services/api_gateway/load_tests -cargo run -r --bin load_test_runner -- normal --num-clients 100 --duration-secs 30 - -# 5. Check results -firefox normal_load_report.html -``` - -## Expected Results - -### Good Performance โœ… -``` -Requests/Second: >100,000 -Error Rate: <0.1% -P99 Latency: <10ฮผs -``` - -### Needs Work โš ๏ธ -``` -Requests/Second: 1,000-10,000 -Error Rate: 0.5%-5% -P99 Latency: 10-100ms -``` - -## Files Generated - -``` -normal_load_report.html # Main dashboard -normal_load_report.rps.svg # RPS chart -normal_load_report.latency.svg # Latency chart -normal_load_report.errors.svg # Error chart -``` - -## Custom Parameters - -```bash -# Custom normal load -cargo run -r --bin load_test_runner -- normal \ - --num-clients 500 \ - --duration-secs 120 - -# Custom spike load -cargo run -r --bin load_test_runner -- spike \ - --target-clients 5000 \ - --ramp-up-secs 15 - -# Custom stress test -cargo run -r --bin load_test_runner -- stress \ - --initial-clients 200 \ - --increment 200 \ - --max-p99-latency-ms 100.0 -``` - -## Monitoring Commands - -```bash -# Real-time metrics -watch -n 1 "curl -s http://localhost:50050/metrics | grep requests_total" - -# Docker stats -watch -n 1 "docker stats --no-stream foxhunt-api-gateway" - -# System resources -watch -n 1 "ps aux | grep api_gateway | grep -v grep" -``` diff --git a/testing/api-gateway-load/QUICK_START.md b/testing/api-gateway-load/QUICK_START.md deleted file mode 100644 index ddfb05c9f..000000000 --- a/testing/api-gateway-load/QUICK_START.md +++ /dev/null @@ -1,242 +0,0 @@ -# API Gateway Load Testing - Quick Start Guide - -## Prerequisites - -1. **Docker Services Running**: - ```bash - docker ps | grep -E "(redis|postgres)" - # Should show: api_gateway_test_redis (port 6380) - # Should show: api_gateway_test_postgres (port 5433) - ``` - -2. **API Gateway Service Running**: - ```bash - curl http://localhost:50050/health - # Should return: {"status":"ok"} - ``` - -3. **Load Test Runner Built**: - ```bash - cargo build --release -p api_gateway_load_tests - ``` - -## Quick Test Commands - -### 1. Normal Load Test (1K clients, 60 seconds) -```bash -cd services/api_gateway/load_tests -cargo run --release --bin load_test_runner -- normal -``` - -**Output**: `normal_load_report.html` + 3 SVG charts - -### 2. Spike Load Test (0โ†’10K clients, 70 seconds) -```bash -cargo run --release --bin load_test_runner -- spike -``` - -**Output**: `spike_load_report.html` + 3 SVG charts - -### 3. Stress Test (Find breaking point) -```bash -cargo run --release --bin load_test_runner -- stress -``` - -**Output**: `stress_test_report.html` + 3 SVG charts - -### 4. Sustained Load Test (100 clients, 24 hours) -```bash -# Run in screen/tmux session -screen -S load_test -cargo run --release --bin load_test_runner -- sustained -# Detach: Ctrl+A, D -``` - -**Output**: `sustained_load_report.html` + 3 SVG charts - -### 5. Run All Tests Sequentially -```bash -cargo run --release --bin load_test_runner -- all -``` - -**Output**: All 3 HTML reports + 9 SVG charts - -## Custom Test Parameters - -### Normal Load (Custom) -```bash -cargo run --release --bin load_test_runner -- normal \ - --gateway-url http://localhost:50050 \ - --num-clients 500 \ - --duration-secs 120 -``` - -### Spike Load (Custom) -```bash -cargo run --release --bin load_test_runner -- spike \ - --gateway-url http://localhost:50050 \ - --target-clients 5000 \ - --ramp-up-secs 15 \ - --sustain-secs 90 -``` - -### Stress Test (Custom) -```bash -cargo run --release --bin load_test_runner -- stress \ - --gateway-url http://localhost:50050 \ - --initial-clients 200 \ - --increment 200 \ - --increment-interval-secs 30 \ - --max-p99-latency-ms 100.0 \ - --max-error-rate-pct 10.0 -``` - -## Viewing Reports - -### In Browser -```bash -firefox normal_load_report.html & -firefox spike_load_report.html & -firefox stress_test_report.html & -``` - -### Extract Key Metrics -```bash -# P99 Latency -grep -A1 "P99 Latency" normal_load_report.html | grep "value" - -# Error Rate -grep -A1 "Error Rate" normal_load_report.html | grep "value" - -# Requests/Second -grep -A1 "Requests/Second" normal_load_report.html | grep "value" -``` - -## Performance Targets - -| Metric | Target | How to Verify | -|--------|--------|---------------| -| P99 Latency | <10ฮผs | Check HTML report summary | -| Throughput | >100K req/s | Check HTML report summary | -| Error Rate | <0.1% | Check HTML report summary | -| P50 Latency | <2ฮผs | Check latency statistics table | -| P90 Latency | <5ฮผs | Check latency statistics table | - -## Troubleshooting - -### Test Fails to Connect -```bash -# Check API Gateway is running -curl http://localhost:50050/health - -# Check port binding -netstat -tlnp | grep 50050 - -# Start API Gateway -cargo run --release -p api_gateway -``` - -### High Latency Results -```bash -# Check system load -htop - -# Check API Gateway logs -docker logs foxhunt-api-gateway --tail 100 - -# Check PostgreSQL connections -psql -h localhost -p 5433 -U postgres -c 'SELECT count(*) FROM pg_stat_activity;' -``` - -### High Error Rate -```bash -# Check API Gateway errors -journalctl -u foxhunt-api-gateway -n 100 - -# Check backend service health -curl http://localhost:50051/health # Trading -curl http://localhost:50052/health # Backtesting -curl http://localhost:50053/health # ML Training -``` - -## Monitoring During Tests - -### Real-time Metrics -```bash -# Watch Prometheus metrics -watch -n 1 "curl -s http://localhost:50050/metrics | grep -E '(requests_total|latency)'" - -# Monitor Docker stats -docker stats foxhunt-api-gateway --no-stream -``` - -### System Resources -```bash -# CPU usage -top -p $(pgrep api_gateway) - -# Memory usage -ps aux | grep api_gateway - -# Network usage -iftop -i lo -``` - -## Quick Validation (5 minutes) - -```bash -# 1. Start services -docker-compose up -d redis postgres -cargo run --release -p api_gateway & - -# 2. Wait for health -sleep 5 -curl http://localhost:50050/health - -# 3. Run quick test (100 clients, 30 seconds) -cd services/api_gateway/load_tests -cargo run --release --bin load_test_runner -- normal \ - --num-clients 100 \ - --duration-secs 30 - -# 4. View report -firefox normal_load_report.html -``` - -## Expected Results - -### Good Performance -``` -Total Requests: 6,000,000+ -Requests/Second: 100,000+ -Error Rate: <0.1% -P99 Latency: <10ฮผs -Circuit Breaker: 0 activations -``` - -### Needs Optimization -``` -Total Requests: 60,000 - 600,000 -Requests/Second: 1,000 - 10,000 -Error Rate: 0.5% - 5% -P99 Latency: 10-100ms -Circuit Breaker: May activate -``` - -## Report Files - -After running tests, you'll find: -``` -normal_load_report.html # Main HTML dashboard -normal_load_report.rps.svg # Requests/second chart -normal_load_report.latency.svg # P99 latency chart -normal_load_report.errors.svg # Error rate chart -``` - -## Next Steps - -1. Review HTML reports for performance insights -2. Check "Capacity Recommendation" section for bottlenecks -3. Implement optimization recommendations -4. Re-run tests to validate improvements -5. Document baseline performance for regression testing diff --git a/testing/e2e/E2E_TEST_GUIDE.md b/testing/e2e/E2E_TEST_GUIDE.md deleted file mode 100644 index a73b077c2..000000000 --- a/testing/e2e/E2E_TEST_GUIDE.md +++ /dev/null @@ -1,449 +0,0 @@ -# Foxhunt E2E Test Suite - Comprehensive Guide - -## ๐Ÿ“‹ Overview - -This directory contains comprehensive end-to-end (E2E) integration tests for the Foxhunt HFT Trading System. The tests validate complete workflows across multiple services, ensuring system reliability, performance, and correctness. - -## ๐ŸŽฏ Test Categories - -### 1. **Core Trading Flow Tests** (`full_trading_flow_e2e.rs`) -Complete trading workflow validation: -- โœ… Market data subscription -- โœ… Order submission and validation -- โœ… Risk management checks -- โœ… Order execution and fills -- โœ… Position updates -- โœ… P&L calculation -- โœ… Account balance updates -- โœ… Order lifecycle with cancellation -- โœ… Risk limit enforcement - -**Key Tests:** -- `test_complete_trading_workflow` - Full end-to-end trading flow -- `test_order_lifecycle_with_cancellation` - Order management -- `test_risk_limit_enforcement` - Risk controls - -### 2. **ML Inference Tests** (`ml_inference_e2e.rs`) -Machine learning model integration: -- โœ… Market data โ†’ feature extraction -- โœ… Real-time model inference (DQN, PPO, MAMBA, TFT, TLOB) -- โœ… Ensemble prediction aggregation -- โœ… Trading signal generation -- โœ… Model performance monitoring -- โœ… Prediction accuracy validation -- โœ… Batch vs streaming consistency - -**Key Tests:** -- `test_complete_ml_inference_pipeline` - Full ML pipeline -- `test_ml_model_failover` - Graceful degradation -- `test_ml_performance_benchmarks` - Performance validation - -### 3. **Risk Management Tests** (`risk_management_e2e.rs`) -Comprehensive risk system validation: -- โœ… VaR (Value at Risk) calculations -- โœ… Position risk assessment -- โœ… Portfolio exposure monitoring -- โœ… Circuit breaker activation -- โœ… Emergency stop functionality -- โœ… Risk alert system -- โœ… Compliance monitoring - -**Key Tests:** -- `test_complete_risk_management_system` - Full risk system -- Portfolio VaR calculation -- Position risk assessment -- Risk metrics validation - -### 4. **Multi-Service Integration** (NEW: `multi_service_integration.rs`) -Cross-service workflow validation: -- โœ… Trading Service + ML Training Service integration -- โœ… Trading Service + Backtesting Service integration -- โœ… Full multi-service data flow -- โœ… Service coordination and communication -- โœ… Configuration transfer between services - -**Key Tests:** -- `test_trading_ml_integration` - Trading + ML coordination -- `test_trading_backtesting_integration` - Trading + Backtesting -- `test_full_multi_service_workflow` - Complete workflow - -### 5. **Error Handling & Recovery** (NEW: `error_handling_recovery.rs`) -System resilience validation: -- โœ… Invalid order handling -- โœ… Service timeout handling -- โœ… ML model failure graceful degradation -- โœ… Concurrent error scenarios -- โœ… Data validation and sanitization - -**Key Tests:** -- `test_invalid_order_handling` - Input validation -- `test_service_timeout_handling` - Timeout management -- `test_ml_model_failure_graceful_degradation` - Failover -- `test_concurrent_error_handling` - Concurrent resilience -- `test_data_validation_and_sanitization` - Edge cases - -### 6. **Performance & Load Tests** (NEW: `performance_load_tests.rs`) -System performance validation: -- โœ… Order submission throughput -- โœ… Concurrent order processing -- โœ… Market data processing throughput -- โœ… ML inference performance -- โœ… Latency percentiles (p50, p95, p99) -- โœ… Sustained load testing - -**Key Tests:** -- `test_order_submission_throughput` - Order rate validation -- `test_concurrent_order_processing` - Concurrent user simulation -- `test_market_data_processing_throughput` - Data pipeline -- `test_ml_inference_performance` - ML performance -- `test_latency_percentiles` - SLA validation -- `test_sustained_load` - Endurance testing - -### 7. **Simplified Integration Tests** (NEW: `simplified_integration_test.rs`) -Basic unit-like integration tests: -- โœ… Type and structure validation -- โœ… Market data structures -- โœ… Order validation logic -- โœ… Risk calculation logic -- โœ… Feature extraction logic -- โœ… Concurrent operations -- โœ… Error handling patterns -- โœ… Data serialization -- โœ… Timestamp handling -- โœ… Collection operations - -**Key Tests:** -- Basic type validation without services -- Standalone logic testing -- No external dependencies required - -## ๐Ÿ—๏ธ Test Infrastructure - -### Core Components - -#### **E2ETestFramework** (`src/framework.rs`) -Main orchestration framework providing: -- Service lifecycle management -- gRPC client connections -- Database testing harness -- ML pipeline testing -- Performance monitoring -- Test data management - -#### **ServiceManager** (`src/services.rs`) -Service orchestration: -- Start/stop all services -- Health monitoring -- Port management -- Process lifecycle - -#### **MLPipelineTestHarness** (`src/ml_pipeline.rs`) -ML testing infrastructure: -- Model health checks -- Feature extraction -- Inference testing -- Ensemble predictions -- Performance metrics - -#### **PerformanceTracker** (`src/performance.rs`) -Performance monitoring: -- Metric recording -- Latency tracking -- Throughput measurement -- Report generation - -### Test Utilities - -#### **Test Data Generation** -- `generate_market_data()` - Realistic market ticks -- `generate_test_order()` - Order generation -- `generate_comprehensive_market_data()` - Multi-symbol data -- `generate_validation_market_data()` - Known patterns - -#### **Helper Functions** -- `wait_for_condition()` - Async condition polling -- Market data processing utilities -- Order validation helpers - -## ๐Ÿš€ Running Tests - -### Run All E2E Tests -```bash -cargo test -p e2e_tests --no-fail-fast -``` - -### Run Specific Test Suite -```bash -# Trading flow tests -cargo test -p e2e_tests --test full_trading_flow_e2e - -# ML inference tests -cargo test -p e2e_tests --test ml_inference_e2e - -# Multi-service integration -cargo test -p e2e_tests --test multi_service_integration - -# Error handling tests -cargo test -p e2e_tests --test error_handling_recovery - -# Performance tests -cargo test -p e2e_tests --test performance_load_tests - -# Simplified tests (no services required) -cargo test -p e2e_tests --test simplified_integration_test -``` - -### Run Specific Test -```bash -cargo test -p e2e_tests --test full_trading_flow_e2e test_complete_trading_workflow -``` - -### Run with Logging -```bash -RUST_LOG=info cargo test -p e2e_tests --test full_trading_flow_e2e -- --nocapture -``` - -### Run in Release Mode (Performance) -```bash -cargo test -p e2e_tests --release --test performance_load_tests -``` - -## ๐Ÿ“Š Test Coverage Summary - -### Existing Tests (Original) -- **Full Trading Flow**: 3 comprehensive tests -- **ML Inference**: 3 model pipeline tests -- **Risk Management**: Complete risk system validation -- **Config Hot Reload**: Configuration management -- **Compliance & Regulatory**: SOX, MiFID II compliance -- **Emergency Shutdown**: Failover scenarios -- **Data Flow Performance**: Pipeline validation -- **Order Lifecycle & Risk**: Combined testing - -**Total Existing**: ~50+ test scenarios across 14 files - -### New Tests Added -1. **Simplified Integration** - 10 basic tests -2. **Multi-Service Integration** - 3 service coordination tests -3. **Error Handling & Recovery** - 5 resilience tests -4. **Performance & Load Tests** - 6 performance tests - -**Total New**: 24 new test scenarios - -### Combined Total -- **~74+ test scenarios** across 18 test files -- **7 major test categories** -- **Complete system coverage** - -## ๐ŸŽฏ Test Objectives - -### Functional Testing -- โœ… Order submission and execution -- โœ… Risk management and compliance -- โœ… ML model inference and predictions -- โœ… Configuration management -- โœ… Data flow and processing - -### Integration Testing -- โœ… Service-to-service communication -- โœ… gRPC API validation -- โœ… Database interactions -- โœ… Multi-service workflows - -### Performance Testing -- โœ… Throughput measurement -- โœ… Latency validation -- โœ… Load testing -- โœ… Concurrent operations -- โœ… Resource utilization - -### Reliability Testing -- โœ… Error handling -- โœ… Failure recovery -- โœ… Graceful degradation -- โœ… Circuit breakers -- โœ… Timeout handling - -## ๐Ÿ“ˆ Performance Targets - -### Latency SLAs -- **p50 (median)**: < 50ms -- **p95**: < 100ms -- **p99**: < 200ms - -### Throughput Targets -- **Order submission**: > 10 orders/sec -- **Market data processing**: > 1,000 ticks/sec -- **ML inference**: < 100ms (batch) - -### Reliability Targets -- **Success rate**: > 95% -- **Uptime**: 99.9% -- **Error recovery**: < 1s - -## ๐Ÿ”ง Test Configuration - -### Environment Variables -```bash -# Service endpoints -TRADING_SERVICE_URL=http://localhost:50051 -BACKTESTING_SERVICE_URL=http://localhost:50052 -ML_TRAINING_SERVICE_URL=http://localhost:50053 - -# Database -DATABASE_URL=postgresql://localhost/foxhunt_test - -# Test settings -E2E_TEST_TIMEOUT=300 # seconds -E2E_LOG_LEVEL=info -``` - -### Test Data -- Market data generated programmatically -- No Redis/Postgres required for basic tests -- Mocks available for offline testing - -## ๐Ÿ› Troubleshooting - -### Common Issues - -#### Test Compilation Errors -```bash -# Clean and rebuild -cargo clean -cargo build -p e2e_tests -``` - -#### Service Connection Failures -- Verify services are running -- Check port availability -- Review service health endpoints - -#### Timeout Issues -- Increase test timeouts -- Check system resources -- Review service logs - -### Debug Mode -```bash -RUST_LOG=debug cargo test -p e2e_tests -- --nocapture -``` - -## ๐Ÿ“ Adding New Tests - -### Basic Structure -```rust -use e2e_tests::{e2e_test, E2ETestFramework}; - -e2e_test!( - test_my_feature, - |mut framework: E2ETestFramework| async { - // Test implementation - Ok(()) - } -); -``` - -### Best Practices -1. Use the `e2e_test!` macro for standardization -2. Record performance metrics -3. Add comprehensive assertions -4. Include cleanup logic -5. Document test purpose and coverage - -## ๐ŸŽ“ Test Patterns - -### Pattern 1: Service Health Check -```rust -let health = framework.check_services_health().await?; -assert!(health.all_healthy); -``` - -### Pattern 2: Client Retrieval -```rust -let trading_client = framework.get_trading_client().await?; -``` - -### Pattern 3: Performance Tracking -```rust -framework.performance_tracker.record_metric("metric_name", value)?; -``` - -### Pattern 4: Error Handling -```rust -match result { - Ok(response) => { /* handle success */ }, - Err(e) => { /* validate error */ } -} -``` - -## ๐Ÿ“Š Test Results - -### Viewing Results -Test results include: -- Pass/fail status -- Execution time -- Performance metrics -- Error details -- Coverage information - -### Metrics Dashboard -Performance metrics are recorded and can be analyzed: -- Latency distributions -- Throughput trends -- Error rates -- Resource utilization - -## ๐Ÿ”ฎ Future Enhancements - -### Planned Additions -1. **Chaos Engineering Tests** - - Random service failures - - Network partition simulation - - Resource exhaustion scenarios - -2. **Extended Performance Tests** - - Soak testing (24+ hours) - - Spike testing - - Stress testing to breaking point - -3. **Security Tests** - - Authentication validation - - Authorization checks - - Input sanitization - - SQL injection prevention - -4. **Compliance Tests** - - Extended regulatory scenarios - - Audit trail validation - - Best execution verification - -## ๐Ÿ“š Related Documentation - -- **Architecture**: `/docs/architecture/` -- **API Documentation**: `/docs/api/` -- **Deployment Guide**: `/docs/deployment/` -- **Monitoring Guide**: `/docs/monitoring/` - -## ๐Ÿค Contributing - -When adding new tests: -1. Follow existing patterns -2. Add documentation -3. Update this guide -4. Include performance metrics -5. Test locally before committing - -## ๐Ÿ“ž Support - -For issues or questions: -- Review test output logs -- Check service status -- Consult architecture documentation -- Review related test files - ---- - -**Last Updated**: 2025-10-01 -**Test Coverage**: 74+ scenarios across 18 files -**Status**: Active Development diff --git a/testing/integration/e2e_helpers/INFRASTRUCTURE_VALIDATION.md b/testing/integration/e2e_helpers/INFRASTRUCTURE_VALIDATION.md deleted file mode 100644 index def2b2dca..000000000 --- a/testing/integration/e2e_helpers/INFRASTRUCTURE_VALIDATION.md +++ /dev/null @@ -1,350 +0,0 @@ -# Infrastructure Health Check Validation - -## Overview - -The `infrastructure_health_check.sh` script validates all Docker Compose infrastructure services required for the Foxhunt HFT trading system. This script is designed to be run by other agents and developers before deploying or testing services. - -## Usage - -```bash -# Run from project root -./tests/e2e_helpers/infrastructure_health_check.sh - -# Check exit code -echo $? -# 0 = all services healthy -# 1 = critical services down (BLOCKER) -# 2 = optional services down (WARNING) -``` - -## Services Validated - -### Critical Infrastructure (Exit 1 if down) - -1. **PostgreSQL (TimescaleDB)** - Port 5432 - - Container: `foxhunt-postgres` - - Checks: Container health + connection test - - Required for: All services (authentication, orders, positions, state) - -2. **Redis** - Port 6379 - - Container: `foxhunt-redis` - - Checks: Container health + PING command - - Required for: Caching, session management, rate limiting - -3. **Vault** - Port 8200 - - Container: `foxhunt-vault` - - Checks: Container health + initialized + unsealed status - - Required for: Secrets management, API keys, credentials - -### Monitoring Infrastructure (Exit 2 if down) - -4. **Prometheus** - Port 9090 - - Container: `foxhunt-prometheus` - - Checks: Container health + HTTP endpoint - - Used for: Metrics collection, alerting - -5. **Grafana** - Port 3000 - - Container: `foxhunt-grafana` - - Checks: Container health + API health endpoint - - Used for: Dashboards, visualization - - Credentials: admin / foxhunt123 - -6. **InfluxDB** - Port 8086 (OPTIONAL) - - Container: `foxhunt-influxdb` - - Checks: Container health + HTTP endpoint - - Used for: Time-series metrics (not currently deployed) - - Status: Defined in docker-compose.yml but not started - -### Application Services (Exit 2 if down) - -7. **API Gateway** - Port 50051 - - Container: `foxhunt-api-gateway` - - Checks: Container health - - Purpose: Authentication, routing, rate limiting - -8. **Trading Service** - Port 50052 - - Container: `foxhunt-trading-service` - - Checks: Container health - - Purpose: Core trading logic, order execution - -9. **Backtesting Service** - Port 50053 - - Container: `foxhunt-backtesting-service` - - Checks: Container health - - Purpose: Strategy testing, performance analytics - -10. **ML Training Service** - Port 50054 - - Container: `foxhunt-ml-training-service` - - Checks: Container health - - Purpose: Model training, feature engineering - -### Storage Services (Exit 2 if down) - -11. **MinIO** - Ports 9000 (API), 9001 (Console) - - Container: `foxhunt-minio` - - Checks: Container health + HTTP endpoint - - Purpose: S3-compatible storage for E2E testing - - Credentials: foxhunt_test / foxhunt_test_password - -## Validation Results - -### Current Status (2025-10-12) - -``` -โœ“ All 17 checks passed -โœ“ Zero critical failures -โœ“ Zero optional failures -โœ“ System ready for deployment -``` - -### Service Health Matrix - -| Service | Status | Port | Container Health | Connectivity | -|---------|--------|------|------------------|--------------| -| PostgreSQL | โœ… HEALTHY | 5432 | โœ… healthy | โœ… OK | -| Redis | โœ… HEALTHY | 6379 | โœ… healthy | โœ… PONG | -| Vault | โœ… HEALTHY | 8200 | โœ… healthy | โœ… unsealed | -| Prometheus | โœ… HEALTHY | 9090 | โœ… healthy | โœ… OK | -| Grafana | โœ… HEALTHY | 3000 | โœ… healthy | โœ… API OK | -| InfluxDB | โš ๏ธ NOT DEPLOYED | 8086 | N/A | N/A | -| API Gateway | โœ… HEALTHY | 50051 | โœ… healthy | N/A | -| Trading Service | โœ… HEALTHY | 50052 | โœ… healthy | N/A | -| Backtesting Service | โœ… HEALTHY | 50053 | โœ… healthy | N/A | -| ML Training Service | โœ… HEALTHY | 50054 | โœ… healthy | N/A | -| MinIO | โœ… HEALTHY | 9000/9001 | โœ… healthy | โœ… OK | - -## Troubleshooting - -### Critical Service Down - -If a critical service (PostgreSQL, Redis, Vault) is down: - -```bash -# Check logs -docker-compose logs - -# Restart specific service -docker-compose restart - -# Restart all infrastructure -docker-compose restart postgres redis vault - -# Full rebuild if needed -docker-compose down -docker-compose up -d -``` - -### PostgreSQL Connection Failed - -```bash -# Verify container is healthy -docker-compose ps postgres - -# Check PostgreSQL logs -docker-compose logs postgres - -# Test connection manually -psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c 'SELECT 1' - -# Verify migrations -cargo sqlx migrate run -``` - -### Redis Connection Failed - -```bash -# Verify container is healthy -docker-compose ps redis - -# Check Redis logs -docker-compose logs redis - -# Test connection manually -docker exec foxhunt-redis redis-cli ping - -# Check memory usage -docker exec foxhunt-redis redis-cli INFO memory -``` - -### Vault Sealed/Not Initialized - -```bash -# Check Vault status -curl http://localhost:8200/v1/sys/health | jq . - -# Vault is running in dev mode and should auto-unseal -# If sealed, restart Vault -docker-compose restart vault - -# Verify dev token -export VAULT_ADDR=http://localhost:8200 -export VAULT_TOKEN=foxhunt-dev-root -docker exec foxhunt-vault vault status -``` - -### InfluxDB Not Running - -InfluxDB is defined in docker-compose.yml but not currently started. To enable: - -```bash -# Start InfluxDB -docker-compose up -d influxdb - -# Verify health -docker-compose ps influxdb -curl http://localhost:8086/health - -# Access web UI -open http://localhost:8086 -# Credentials: foxhunt / foxhunt_dev_password -``` - -### Application Service Down - -```bash -# Check service logs -docker-compose logs api_gateway -docker-compose logs trading_service -docker-compose logs backtesting_service -docker-compose logs ml_training_service - -# Restart service -docker-compose restart - -# Rebuild service if needed -docker-compose up -d --build -``` - -## Integration with Other Scripts - -This health check script is designed to be used by other testing scripts: - -```bash -#!/bin/bash -# Example: E2E test runner - -# Validate infrastructure first -./tests/e2e_helpers/infrastructure_health_check.sh -HEALTH_EXIT=$? - -if [ $HEALTH_EXIT -eq 1 ]; then - echo "CRITICAL: Infrastructure not ready" - exit 1 -elif [ $HEALTH_EXIT -eq 2 ]; then - echo "WARNING: Some optional services down, continuing..." -fi - -# Proceed with tests -cargo test --test e2e_trading_integration -``` - -## Manual Verification Commands - -### PostgreSQL -```bash -psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -\dt # List tables -\q # Quit -``` - -### Redis -```bash -docker exec -it foxhunt-redis redis-cli -PING # Should return PONG -INFO -QUIT -``` - -### Vault -```bash -curl http://localhost:8200/v1/sys/health | jq . -# Should show: initialized=true, sealed=false -``` - -### Prometheus -```bash -# Check targets -curl http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | {job: .labels.job, health: .health}' - -# Web UI -open http://localhost:9090 -``` - -### Grafana -```bash -# API health -curl http://localhost:3000/api/health | jq . - -# Web UI -open http://localhost:3000 -# Login: admin / foxhunt123 -``` - -### MinIO -```bash -# API health -curl http://localhost:9000/minio/health/live - -# Console UI -open http://localhost:9001 -# Login: foxhunt_test / foxhunt_test_password -``` - -## Performance Benchmarks - -From Wave 137 validation (Agent 314): - -- **Health check duration**: ~2-3 seconds -- **Container health checks**: 11 containers validated -- **Connectivity tests**: 7 services tested -- **Total checks**: 17 validation points -- **Success rate**: 100% (17/17 passed) -- **Zero false positives**: All checks accurate - -## Exit Codes Reference - -| Exit Code | Status | Meaning | Action Required | -|-----------|--------|---------|-----------------| -| 0 | SUCCESS | All services healthy | Proceed with deployment/testing | -| 1 | CRITICAL | Critical services down | Fix PostgreSQL/Redis/Vault before proceeding | -| 2 | WARNING | Optional services down | Can proceed but monitoring limited | - -## Maintenance Notes - -### Adding New Services - -To add a new service to validation: - -1. Determine if service is critical or optional -2. Add check in appropriate section (2, 3, 4, or 5) -3. Use `check_docker_container` for container health -4. Add connectivity test if applicable -5. Update this documentation - -### Modifying Critical Services - -If a service changes from optional to critical: - -1. Change last parameter of `check_docker_container` from "false" to "true" -2. Update documentation in "Services Validated" section -3. Update "Service Health Matrix" table - -### Removing Services - -If a service is deprecated: - -1. Remove check from script -2. Update documentation -3. Update exit code expectations in dependent scripts - -## Related Documentation - -- `/home/jgrusewski/Work/foxhunt/CLAUDE.md` - System architecture and credentials -- `/home/jgrusewski/Work/foxhunt/docker-compose.yml` - Service definitions -- `/home/jgrusewski/Work/foxhunt/WAVE_137_VALIDATION_SUMMARY.md` - E2E validation results - -## Last Updated - -- **Date**: 2025-10-12 -- **Agent**: Agent 314 -- **Wave**: 137 -- **Status**: ALL SERVICES VALIDATED โœ… diff --git a/testing/integration/e2e_helpers/QUICKSTART.md b/testing/integration/e2e_helpers/QUICKSTART.md deleted file mode 100644 index 015d2ad5b..000000000 --- a/testing/integration/e2e_helpers/QUICKSTART.md +++ /dev/null @@ -1,151 +0,0 @@ -# JWT Token Generator - Quick Start - -**5-Minute Guide** for E2E Testing - -## Installation - -โœ… **Already installed** - No additional setup required! - -Dependencies: Python3 + PyJWT (already available) - -## Basic Usage - -### 1. Generate Token (Default) -```bash -cd tests/e2e_helpers -./jwt_token_generator.sh -``` -Output: JWT token for trader with 1-hour expiration - -### 2. Use in API Request -```bash -TOKEN=$(./jwt_token_generator.sh) -curl -H "Authorization: Bearer $TOKEN" http://localhost:50051/api/v1/orders -``` - -### 3. Admin Token -```bash -./jwt_token_generator.sh admin_user admin "api.access,system.admin" -``` - -### 4. Custom Expiration (10 minutes) -```bash -./jwt_token_generator.sh test_user trader "api.access" 600 -``` - -## Common Patterns - -### E2E Test Script -```bash -#!/bin/bash -cd tests/e2e_helpers - -# Generate token -TOKEN=$(./jwt_token_generator.sh) - -# Test endpoint -curl -H "Authorization: Bearer $TOKEN" \ - http://localhost:50051/api/v1/orders - -# Submit order -curl -X POST \ - -H "Authorization: Bearer $TOKEN" \ - -H "Content-Type: application/json" \ - -d '{"symbol": "BTC/USD", "quantity": 1.0, "side": "buy"}' \ - http://localhost:50051/api/v1/orders -``` - -### Load Test -```bash -# Generate 10 tokens -for i in {1..10}; do - TOKEN=$(./jwt_token_generator.sh "user_$i" trader "api.access") - echo "$TOKEN" > "token_$i.txt" -done -``` - -## Command Syntax - -```bash -./jwt_token_generator.sh [user_id] [role] [permissions] [ttl_seconds] -``` - -| Argument | Default | Example | -|----------|---------|---------| -| user_id | test_user_123 | admin_user | -| role | trader | admin, viewer | -| permissions | api.access | "api.access,system.admin" | -| ttl_seconds | 3600 | 600 (10 min) | - -## Role Presets - -```bash -# Trader (default) -./jwt_token_generator.sh - -# Admin -./jwt_token_generator.sh admin_user admin "api.access,system.admin" - -# Viewer (read-only) -./jwt_token_generator.sh viewer_user viewer "api.read" -``` - -## Environment Variable - -```bash -# Custom JWT secret -export JWT_SECRET="your-secret-key-here" -./jwt_token_generator.sh -``` - -## Troubleshooting - -### Token not working? -1. Check API Gateway is running: `curl http://localhost:8080/health` -2. Verify JWT secret matches: Check `.env` file -3. Regenerate token: `TOKEN=$(./jwt_token_generator.sh)` - -### Permission denied? -Generate admin token: -```bash -./jwt_token_generator.sh admin admin "api.access,system.admin" -``` - -### Token expired? -Use shorter expiration for testing: -```bash -./jwt_token_generator.sh user trader "api.access" 60 # 60 seconds -``` - -## Documentation - -- **Full docs**: [README.md](README.md) -- **Examples**: [USAGE_EXAMPLES.md](USAGE_EXAMPLES.md) -- **Validation**: [VALIDATION_REPORT.md](VALIDATION_REPORT.md) - -## Quick Reference - -```bash -# Default token (trader, 1 hour) -./jwt_token_generator.sh - -# Admin token -./jwt_token_generator.sh admin_user admin "api.access,system.admin" - -# Short-lived (60 sec) -./jwt_token_generator.sh user trader "api.access" 60 - -# Multiple permissions -./jwt_token_generator.sh user trader "api.access,orders.submit,risk.view" - -# Custom secret -JWT_SECRET="secret" ./jwt_token_generator.sh - -# Inspect token -TOKEN=$(./jwt_token_generator.sh) -python3 -c "import jwt, json; token='$TOKEN'; print(json.dumps(jwt.decode(token, options={'verify_signature': False}), indent=2))" -``` - ---- - -**Ready to use!** Run `./jwt_token_generator.sh` to get started. diff --git a/testing/integration/e2e_helpers/QUICK_START_INFRASTRUCTURE.md b/testing/integration/e2e_helpers/QUICK_START_INFRASTRUCTURE.md deleted file mode 100644 index 4801ebbcc..000000000 --- a/testing/integration/e2e_helpers/QUICK_START_INFRASTRUCTURE.md +++ /dev/null @@ -1,168 +0,0 @@ -# Quick Start: Infrastructure Validation - -**For Agents and Developers**: Fast reference guide for infrastructure validation - ---- - -## TL;DR - One Command - -```bash -# Validate all infrastructure (from project root) -./tests/e2e_helpers/infrastructure_health_check.sh - -# Check exit code -echo $? -# 0 = All healthy, proceed -# 1 = Critical failure, DO NOT PROCEED -# 2 = Optional failure, can proceed with warnings -``` - ---- - -## Service Status Quick Check - -```bash -# All services at a glance -docker-compose ps - -# Expected output: All services "Up (healthy)" -``` - ---- - -## Critical Services (MUST BE HEALTHY) - -### PostgreSQL - Port 5432 -```bash -# Quick test -psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c 'SELECT 1' - -# Should return: (1 row) -``` - -### Redis - Port 6379 -```bash -# Quick test -docker exec foxhunt-redis redis-cli ping - -# Should return: PONG -``` - -### Vault - Port 8200 -```bash -# Quick test -curl -s http://localhost:8200/v1/sys/health | jq '.sealed' - -# Should return: false -``` - ---- - -## Application Services - -All should be "Up (healthy)" in `docker-compose ps`: - -- **API Gateway**: Port 50051 -- **Trading Service**: Port 50052 -- **Backtesting Service**: Port 50053 -- **ML Training Service**: Port 50054 - ---- - -## Common Issues - -### Service Not Healthy - -```bash -# Restart single service -docker-compose restart - -# View logs -docker-compose logs -f -``` - -### All Services Down - -```bash -# Restart all services -docker-compose down -docker-compose up -d - -# Wait 30-60 seconds for health checks -docker-compose ps -``` - -### PostgreSQL Connection Failed - -```bash -# Check if running -docker-compose ps postgres - -# Restart -docker-compose restart postgres - -# Verify migrations -cd /home/jgrusewski/Work/foxhunt -cargo sqlx migrate run -``` - ---- - -## Pre-Test Checklist - -Before running any E2E tests: - -- [ ] Run `./tests/e2e_helpers/infrastructure_health_check.sh` -- [ ] Exit code = 0 (all healthy) -- [ ] PostgreSQL responding to queries -- [ ] Redis responding to PING -- [ ] Vault unsealed - ---- - -## URLs Quick Reference - -| Service | URL | Credentials | -|---------|-----|-------------| -| PostgreSQL | postgresql://localhost:5432/foxhunt | foxhunt / foxhunt_dev_password | -| Redis | redis://localhost:6379 | (no auth) | -| Vault | http://localhost:8200 | Token: foxhunt-dev-root | -| Prometheus | http://localhost:9090 | (no auth) | -| Grafana | http://localhost:3000 | admin / foxhunt123 | -| MinIO Console | http://localhost:9001 | foxhunt_test / foxhunt_test_password | - ---- - -## Integration Example - -```bash -#!/bin/bash -# Example E2E test runner - -echo "Validating infrastructure..." -./tests/e2e_helpers/infrastructure_health_check.sh -EXIT_CODE=$? - -if [ $EXIT_CODE -eq 1 ]; then - echo "ERROR: Critical services down, cannot proceed" - exit 1 -elif [ $EXIT_CODE -eq 2 ]; then - echo "WARNING: Optional services down, continuing..." -fi - -echo "Running E2E tests..." -cargo test --test e2e_trading_integration -``` - ---- - -## Full Documentation - -For comprehensive details, see: -- `INFRASTRUCTURE_VALIDATION.md` - Complete validation guide -- `AGENT_314_INFRASTRUCTURE_VALIDATION_REPORT.md` - Validation results - ---- - -**Last Validated**: 2025-10-12 (Agent 314, Wave 137) -**Status**: โœ… 100% HEALTHY (17/17 checks passed) diff --git a/testing/integration/e2e_helpers/USAGE_EXAMPLES.md b/testing/integration/e2e_helpers/USAGE_EXAMPLES.md deleted file mode 100644 index c57ed826b..000000000 --- a/testing/integration/e2e_helpers/USAGE_EXAMPLES.md +++ /dev/null @@ -1,207 +0,0 @@ -# JWT Token Generator - Usage Examples - -Quick reference for common JWT token generation scenarios in E2E testing. - -## Basic Usage - -### Generate Default Token -```bash -./jwt_token_generator.sh -``` -Output: Trader token with `api.access` permission, 1-hour expiration - -### Use in Curl Request -```bash -TOKEN=$(./jwt_token_generator.sh) -curl -H "Authorization: Bearer $TOKEN" http://localhost:50051/api/v1/orders -``` - -## Role-Based Scenarios - -### Trader (Default) -```bash -./jwt_token_generator.sh -# or -./jwt_token_generator.sh trader_user trader "api.access" -``` - -### Admin User -```bash -./jwt_token_generator.sh admin_user admin "api.access,system.admin,config.write" -``` - -### Viewer (Read-Only) -```bash -./jwt_token_generator.sh viewer_user viewer "api.read,data.read,metrics.view" -``` - -## Custom Expiration - -### Short-Lived (60 seconds) -```bash -./jwt_token_generator.sh test_user trader "api.access" 60 -``` - -### Long-Lived (24 hours) -```bash -./jwt_token_generator.sh test_user trader "api.access" 86400 -``` - -### Very Short (10 seconds, for expiration testing) -```bash -./jwt_token_generator.sh test_user trader "api.access" 10 -``` - -## Permission Combinations - -### Trading Permissions -```bash -./jwt_token_generator.sh trader_user trader "api.access,orders.submit,orders.cancel,positions.view" -``` - -### Risk Management -```bash -./jwt_token_generator.sh risk_user risk_manager "api.access,risk.view,risk.limits,circuit.breaker" -``` - -### System Administration -```bash -./jwt_token_generator.sh admin_user admin "api.access,system.admin,config.write,config.reload,users.manage" -``` - -### Monitoring Only -```bash -./jwt_token_generator.sh monitor_user viewer "api.access,metrics.view,health.check,alerts.view" -``` - -## Integration Test Scripts - -### Order Submission Test -```bash -#!/bin/bash -TOKEN=$(./jwt_token_generator.sh) - -curl -X POST \ - -H "Authorization: Bearer $TOKEN" \ - -H "Content-Type: application/json" \ - -d '{"symbol": "BTC/USD", "quantity": 1.0, "side": "buy", "price": 50000.0}' \ - http://localhost:50051/api/v1/orders -``` - -### Position Query Test -```bash -#!/bin/bash -TOKEN=$(./jwt_token_generator.sh) - -curl -H "Authorization: Bearer $TOKEN" \ - http://localhost:50051/api/v1/positions -``` - -### Config Management Test (Admin) -```bash -#!/bin/bash -ADMIN_TOKEN=$(./jwt_token_generator.sh admin_user admin "api.access,config.write") - -curl -X POST \ - -H "Authorization: Bearer $ADMIN_TOKEN" \ - -H "Content-Type: application/json" \ - -d '{"key": "max_order_size", "value": "100.0"}' \ - http://localhost:50051/api/v1/config -``` - -## Load Testing - -### Generate Multiple User Tokens -```bash -#!/bin/bash -# Generate 100 unique user tokens -for i in {1..100}; do - TOKEN=$(./jwt_token_generator.sh "user_$i" trader "api.access") - echo "$TOKEN" > "tokens/token_$i.txt" -done -``` - -### Parallel Requests -```bash -#!/bin/bash -# Use different tokens for concurrent requests -for i in {1..10}; do - TOKEN=$(cat "tokens/token_$i.txt") - curl -H "Authorization: Bearer $TOKEN" \ - http://localhost:50051/api/v1/orders & -done -wait -``` - -## Token Expiration Testing - -### Test Expired Token -```bash -#!/bin/bash -# Generate 5-second token -TOKEN=$(./jwt_token_generator.sh test_user trader "api.access" 5) - -# Use immediately (should succeed) -curl -H "Authorization: Bearer $TOKEN" http://localhost:50051/api/v1/orders -echo "First request: OK" - -# Wait for expiration -sleep 6 - -# Use again (should fail with 401) -curl -H "Authorization: Bearer $TOKEN" http://localhost:50051/api/v1/orders -echo "Second request: Should fail with 401" -``` - -### Test Token Refresh -```bash -#!/bin/bash -# Old token (about to expire) -OLD_TOKEN=$(./jwt_token_generator.sh user1 trader "api.access" 10) - -# New token (fresh) -NEW_TOKEN=$(./jwt_token_generator.sh user1 trader "api.access" 3600) - -# Use old token -curl -H "Authorization: Bearer $OLD_TOKEN" http://localhost:50051/api/v1/orders - -# Wait for old token to expire -sleep 11 - -# Use new token (should work) -curl -H "Authorization: Bearer $NEW_TOKEN" http://localhost:50051/api/v1/orders -``` - -## Environment Variable Configuration - -### Production Secret -```bash -export JWT_SECRET="your-production-secret-key-here-must-be-long-enough-for-security" -TOKEN=$(./jwt_token_generator.sh) -``` - -### Multiple Environments -```bash -# Development -JWT_SECRET="dev-secret-123" DEV_TOKEN=$(./jwt_token_generator.sh) - -# Staging -JWT_SECRET="staging-secret-456" STAGING_TOKEN=$(./jwt_token_generator.sh) - -# Production -JWT_SECRET="prod-secret-789" PROD_TOKEN=$(./jwt_token_generator.sh) -``` - -## Debugging - -### Inspect Token Claims -```bash -TOKEN=$(./jwt_token_generator.sh) - -# Decode token (requires python3 and pyjwt) -python3 <`) - -### Performance Metrics - -- **First load**: ~5-10ms (421 bars from DBN file) -- **Subsequent loads**: <1ms (from cache) -- **Memory overhead**: ~50KB per cached symbol - ---- - -## Testing Instructions - -### Run All Trading Service E2E Tests - -```bash -# With services running (docker-compose up -d) -cargo test -p integration_tests --test trading_service_e2e - -# Expected: All 15 tests pass with real DBN data -``` - -### Run Specific Test with DBN Data - -```bash -# Test market order submission -cargo test -p integration_tests --test trading_service_e2e \ - test_e2e_order_submission_market_order -- --nocapture - -# Test with realistic limit price -cargo test -p integration_tests --test trading_service_e2e \ - test_e2e_order_submission_limit_order -- --nocapture -``` - -### Verify DBN Data Loading - -```bash -# Run DBN helper tests -cargo test -p integration_tests dbn_helpers::tests -- --nocapture -``` - ---- - -## Troubleshooting - -### Issue: DBN File Not Found - -**Symptom**: Test fails with "DBN test data not found" - -**Solution**: -```bash -# Verify test data exists -ls -lh test_data/real/databento/ES.FUT_ohlcv-1m_2024-01-02.dbn - -# Check workspace root detection -cd services/integration_tests -cargo test dbn_helpers::tests::test_dbn_manager_creation -- --nocapture -``` - -### Issue: Unrealistic Prices - -**Symptom**: Order prices seem wrong for ES.FUT - -**Expected Range**: $4,700 - $4,770 (January 2024 data) - -**Check**: -```rust -// Verify data loading -let manager = DbnTestDataManager::new().await?; -let price = manager.get_realistic_price("ES.FUT").await?; -println!("Current ES.FUT price: ${:.2}", price); // Should be ~$4,750 -``` - -### Issue: Test Timeouts - -**Symptom**: Tests timeout waiting for market data events - -**Expected Behavior**: Tests should handle timeout gracefully and pass even without live market data. The `test_e2e_market_data_subscription` test uses a 2-second timeout and passes whether or not events are received. - ---- - -## Future Enhancements - -### 1. Additional Symbols - -Add more DBN files for different instruments: - -```rust -// NQ.FUT (Nasdaq futures) -file_mapping.insert("NQ.FUT", "test_data/real/databento/NQ.FUT_ohlcv-1m.dbn"); - -// CL.FUT (Crude oil futures) -file_mapping.insert("CL.FUT", "test_data/real/databento/CL.FUT_ohlcv-1m.dbn"); -``` - -### 2. Multi-Symbol Testing - -Test cross-symbol scenarios: - -```rust -// Test ES.FUT vs NQ.FUT correlation -let dbn_manager = get_dbn_manager().await?; -let es_bars = dbn_manager.get_last_n_bars("ES.FUT", 10).await?; -let nq_bars = dbn_manager.get_last_n_bars("NQ.FUT", 10).await?; -``` - -### 3. Historical Replay - -Implement time-based replay for backtesting: - -```rust -// Replay specific time window -let (start, end) = dbn_manager.get_time_range("ES.FUT").await?; -let window = dbn_manager.get_data_window( - "ES.FUT", - start, - start + Duration::hours(1) -).await?; -``` - -### 4. Market Condition Testing - -Test different market regimes: - -- **High volatility**: Market open, FOMC announcements -- **Low volatility**: Overnight sessions -- **Trend following**: Strong directional moves -- **Mean reversion**: Range-bound periods - ---- - -## Related Files - -### Core Implementation - -- `services/backtesting_service/src/dbn_data_source.rs`: DBN file loading -- `services/backtesting_service/src/dbn_repository.rs`: Repository pattern -- `data/src/providers/databento/dbn_parser.rs`: Zero-copy DBN parsing - -### Test Infrastructure - -- `services/integration_tests/tests/common/dbn_helpers.rs`: Helper functions -- `services/integration_tests/tests/trading_service_e2e.rs`: E2E test suite -- `test_data/real/databento/`: DBN data files - -### Documentation - -- `TESTING_PLAN.md`: Overall testing strategy -- `services/backtesting_service/README.md`: Backtesting service docs - ---- - -## Validation Checklist - -- [x] DBN dependencies added to `Cargo.toml` -- [x] `DbnTestDataManager` helper created -- [x] All 15 E2E tests updated to use ES.FUT -- [x] Realistic price calculation implemented -- [x] Symbol mapping documented -- [x] Test output enhanced with DBN annotations -- [x] Caching strategy implemented -- [x] Error handling for missing files -- [x] Helper tests added -- [ ] Full test suite validation (pending build) - ---- - -## Conclusion - -The integration of real DBN market data into trading service E2E tests provides: - -1. **Higher fidelity testing** with production-like data -2. **Better coverage** of real-world scenarios -3. **Easier maintenance** through code reuse -4. **Future extensibility** for additional symbols and timeframes - -All tests now use **ES.FUT** with realistic prices from January 2, 2024, providing a solid foundation for reliable E2E testing of the trading service. - ---- - -**Next Steps**: -1. Validate all tests pass with real data -2. Add more symbols (NQ.FUT, CL.FUT) as needed -3. Consider adding different time periods for regime testing -4. Document any symbol-specific behavior discovered during testing