foxhunt/scripts/README_test_tli_tuning.md

TLI Tuning Workflow Test Script

Script: /home/jgrusewski/Work/foxhunt/scripts/test_tli_tuning.sh Purpose: End-to-end testing of the TLI hyperparameter tuning workflow Version: 1.0.0 Last Updated: 2025-10-13

---

Overview

This script provides comprehensive testing of the TLI hyperparameter tuning functionality, including:

• Prerequisite validation (services, authentication, data)
• Job submission and tracking
• Status polling with real-time progress
• Best parameter retrieval and export
• Graceful error handling and cleanup

Features

1. Prerequisite Checks ✅

• TLI binary existence and location
• JWT token validation (with masking for security)
• API Gateway health (HTTP + gRPC)
• ML Training Service availability
• Config file creation (auto-generates if missing)
• Test data validation with size reporting
• Optional tool detection (grpcurl, jq)

2. Three Operating Modes

Full Workflow (Default)

./scripts/test_tli_tuning.sh

Steps:

1. Check prerequisites
2. Start tuning job (DQN, 5 trials)
3. Poll status every 5 seconds
4. Display real-time progress with timestamps
5. Retrieve best parameters when complete
6. Export results to YAML file

Duration: ~5-10 minutes (depending on trials)

Quick Test Mode

./scripts/test_tli_tuning.sh --quick

Steps:

1. Check prerequisites
2. Start tuning job
3. Display job ID and monitoring commands
4. Exit immediately (no polling)

Duration: ~10 seconds Use Case: Verify job submission works without waiting

Check-Only Mode

./scripts/test_tli_tuning.sh --check-only

Steps:

1. Validate all prerequisites
2. Report status of services/files
3. Exit with detailed diagnostics

Duration: ~5 seconds Use Case: Pre-flight checks before running tests

3. Customization Options

# Custom model type
./scripts/test_tli_tuning.sh --model PPO --trials 10

# Environment variable overrides
export POLL_INTERVAL=10         # Poll every 10 seconds
export MAX_WAIT_TIME=600        # Wait up to 10 minutes
export TLI_BIN=/custom/path/tli # Custom TLI location
./scripts/test_tli_tuning.sh

4. Error Handling

• Automatic cleanup: Stops jobs on script failure/interrupt
• Exit traps: SIGINT, SIGTERM handled gracefully
• Detailed diagnostics: Clear error messages with remediation steps
• Progress tracking: Job IDs saved to ~/.foxhunt/test_tuning_job.txt

5. Output Features

• Color-coded status: ✅ Green (success), ❌ Red (error), ⚠️ Yellow (warning)
• Real-time progress: Timestamps, trial progress, elapsed time
• Progress visualization: Text-based progress bar
• Duration tracking: Total test duration reported
• Masked credentials: JWT tokens masked for security

---

Prerequisites

1. Build TLI Binary

cargo build --release -p tli
# Binary: target/release/tli

2. Authenticate with TLI

./target/release/tli auth login --username <user> --password <pass>
# Creates: ~/.foxhunt/jwt_token

3. Start Infrastructure Services

docker-compose up -d
# Services: postgres, redis, vault, influxdb

4. Start API Gateway

cargo run --release -p api_gateway
# Ports: 50051 (gRPC), 8080 (HTTP health)

5. Start ML Training Service

cargo run --release -p ml_training_service
# Ports: 50054 (gRPC), 8095 (HTTP health)

6. Verify Health

./scripts/comprehensive_health_check.sh
# Should show all services healthy

---

Usage Examples

Example 1: Basic Full Workflow Test

$ ./scripts/test_tli_tuning.sh

================================================================================
  Prerequisite Checks
================================================================================

[STEP] Checking TLI binary...
✓ TLI binary found
ℹ Location: /home/jgrusewski/Work/foxhunt/target/release/tli

[STEP] Checking JWT token...
✓ JWT token found
ℹ Token (masked): eyJhbGciOiJIUzI1NiIs...WXZ6aGJHVnU=

[STEP] Checking API Gateway health...
✓ API Gateway is healthy

[STEP] Checking ML Training Service health...
✓ ML Training Service is healthy

[STEP] Checking tuning config file...
✓ Config file found
ℹ Path: /home/jgrusewski/Work/foxhunt/test_data/tuning_config.yaml

[STEP] Checking test data...
✓ Test data found
ℹ Path: /home/jgrusewski/Work/foxhunt/test_data/btcusdt_sample_100.parquet
ℹ Size: 1.2M

✓ All prerequisites satisfied

================================================================================
  Starting Tuning Job
================================================================================

[STEP] Submitting tuning job to TLI...
ℹ Command: ./target/release/tli tune start --model DQN --trials 5 --config test_data/tuning_config.yaml

🚀 Starting hyperparameter tuning job...
   Model: DQN
   Trials: 5
   Config: test_data/tuning_config.yaml
   GPU: ❌ Disabled

✅ Tuning job started successfully!
   Job ID: 550e8400-e29b-41d4-a716-446655440000
   Saved to ~/.foxhunt/tuning_jobs.json

================================================================================
  Polling Job Status
================================================================================

ℹ Polling every 5s (max wait: 5m)

[14:30:05] Status: RUNNING | Progress: 0/5 (0.0%) | Elapsed: 0s
[14:30:10] Status: RUNNING | Progress: 1/5 (20.0%) | Elapsed: 5s
[14:30:15] Status: RUNNING | Progress: 2/5 (40.0%) | Elapsed: 10s
[14:30:20] Status: RUNNING | Progress: 3/5 (60.0%) | Elapsed: 15s
[14:30:25] Status: RUNNING | Progress: 4/5 (80.0%) | Elapsed: 20s
[14:30:30] Status: RUNNING | Progress: 5/5 (100.0%) | Elapsed: 25s

✅ Job completed successfully!

📊 Tuning Job Status
   Status: TUNING_COMPLETED
   Progress: 5/5 trials (100.0%)
   [██████████████████████████████████████████████████] 100.0%

🏆 Best Results So Far
   Sharpe Ratio: 2.3456
   Elapsed Time: 30 seconds

================================================================================
  Retrieving Best Parameters
================================================================================

[STEP] Fetching best hyperparameters...

🏆 Best Performance Metrics
   sharpe_ratio: 2.3456
   total_return: 15.6789
   max_drawdown: 8.4321

📋 Best Hyperparameters
┌──────────────────┬──────────┬───────────────┐
│ Parameter        │ Value    │ Type          │
├──────────────────┼──────────┼───────────────┤
│ learning_rate    │ 0.000512 │ Learning Rate │
│ batch_size       │ 64.000000│ Integer       │
│ gamma            │ 0.976543 │ Float         │
│ epsilon_decay    │ 0.995678 │ Float         │
└──────────────────┴──────────┴───────────────┘

[STEP] Exporting best parameters to file...
✅ Parameters exported to: best_params_550e8400.yaml

💡 Use these parameters in your training configuration.

================================================================================
  Test Completed Successfully
================================================================================

✅ Full workflow executed without errors
ℹ Total duration: 35s

Example 2: Quick Test (No Waiting)

$ ./scripts/test_tli_tuning.sh --quick

================================================================================
  Quick TLI Tuning Test (Start Only)
================================================================================

[Prerequisites checks...]
✅ All prerequisites satisfied

================================================================================
  Starting Tuning Job
================================================================================

✅ Tuning job started successfully!
   Job ID: 550e8400-e29b-41d4-a716-446655440000

✅ Job started successfully
ℹ Monitor with: ./target/release/tli tune status --job-id 550e8400-e29b-41d4-a716-446655440000
ℹ Get results: ./target/release/tli tune best --job-id 550e8400-e29b-41d4-a716-446655440000
ℹ Stop job: ./target/release/tli tune stop --job-id 550e8400-e29b-41d4-a716-446655440000
ℹ Total test duration: 8s

Example 3: Custom Model and Trials

$ ./scripts/test_tli_tuning.sh --model MAMBA_2 --trials 10

[Prerequisites checks...]
✅ All prerequisites satisfied

🚀 Starting hyperparameter tuning job...
   Model: MAMBA_2
   Trials: 10
   Config: test_data/tuning_config.yaml
   GPU: ❌ Disabled

[Polling progress 0-100%...]

Example 4: Check Prerequisites Only

$ ./scripts/test_tli_tuning.sh --check-only

================================================================================
  Prerequisite Checks
================================================================================

[STEP] Checking TLI binary...
✓ TLI binary found
ℹ Location: /home/jgrusewski/Work/foxhunt/target/release/tli

[STEP] Checking JWT token...
✗ JWT token not found at: /home/jgrusewski/.foxhunt/jwt_token
ℹ Authenticate with: ./target/release/tli auth login

[STEP] Checking API Gateway health...
✗ API Gateway not responding at http://localhost:8080/health
ℹ Start services with: docker-compose up -d

[STEP] Checking ML Training Service health...
✓ ML Training Service is healthy

✗ Prerequisites check FAILED
ℹ Please resolve issues above and retry

---

Configuration

Environment Variables

Variable	Default	Description
TLI_BIN	target/release/tli	Path to TLI binary
JWT_TOKEN_FILE	~/.foxhunt/jwt_token	JWT token location
POLL_INTERVAL	5	Status polling interval (seconds)
MAX_WAIT_TIME	300	Maximum wait time (seconds)

Generated Files

File	Location	Purpose
JWT Token	~/.foxhunt/jwt_token	Authentication
Tuning Jobs	~/.foxhunt/tuning_jobs.json	Job tracking
Test Job ID	~/.foxhunt/test_tuning_job.txt	Current test job
Minimal Config	test_data/tuning_config.yaml	Auto-generated config
Best Params	best_params_<job_id>.yaml	Exported parameters

Minimal Config Structure

The script auto-generates a minimal config if none exists:

# test_data/tuning_config.yaml
tuning:
  study_name: "test_study"
  storage: "sqlite:///optuna_test.db"
  direction: "maximize"  # Maximize Sharpe ratio

  search_space:
    learning_rate:
      type: "float"
      low: 0.0001
      high: 0.01
      log: true

    batch_size:
      type: "int"
      low: 32
      high: 128
      step: 32

    gamma:
      type: "float"
      low: 0.90
      high: 0.99

    epsilon_decay:
      type: "float"
      low: 0.990
      high: 0.999

  training:
    epochs: 10
    validation_split: 0.2
    early_stopping_patience: 3

  metrics:
    - "sharpe_ratio"
    - "total_return"
    - "max_drawdown"
    - "win_rate"

---

Troubleshooting

Issue: TLI Binary Not Found

Error:

✗ TLI binary not found at: target/release/tli
ℹ Build with: cargo build --release -p tli

Solution:

cargo build --release -p tli

---

Issue: JWT Token Missing

Error:

✗ JWT token not found at: ~/.foxhunt/jwt_token
ℹ Authenticate with: ./target/release/tli auth login

Solution:

./target/release/tli auth login --username admin --password admin123

Note: Default dev credentials from docker-compose.yml

---

Issue: API Gateway Not Running

Error:

✗ API Gateway not responding at http://localhost:8080/health
ℹ Start services with: docker-compose up -d

Solution:

# Start infrastructure
docker-compose up -d

# Start API Gateway
cargo run --release -p api_gateway

---

Issue: ML Training Service Unavailable

Error:

✗ ML Training Service not responding at http://localhost:8095/health

Solution:

# Check service status
curl http://localhost:8095/health

# Start if not running
cargo run --release -p ml_training_service

# Check logs
docker-compose logs ml_training_service

---

Issue: Job Times Out

Symptom: Job exceeds MAX_WAIT_TIME (300s default)

Solution:

# Increase max wait time
export MAX_WAIT_TIME=600  # 10 minutes
./scripts/test_tli_tuning.sh

# Or reduce trials for faster completion
./scripts/test_tli_tuning.sh --trials 3

---

Issue: Config File Not Found

Symptom: Warning about missing config file

Behavior: Script auto-creates minimal config at test_data/tuning_config.yaml

Verification:

cat test_data/tuning_config.yaml

---

Integration with CI/CD

GitHub Actions Example

name: Test TLI Tuning
on: [push, pull_request]

jobs:
  test-tli-tuning:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2

      - name: Start infrastructure
        run: docker-compose up -d

      - name: Build TLI
        run: cargo build --release -p tli

      - name: Start services
        run: |
          cargo run --release -p api_gateway &
          cargo run --release -p ml_training_service &
          sleep 10

      - name: Authenticate TLI
        run: |
          ./target/release/tli auth login \
            --username admin \
            --password ${{ secrets.TLI_PASSWORD }}

      - name: Run quick test
        run: ./scripts/test_tli_tuning.sh --quick

      - name: Check prerequisites
        run: ./scripts/test_tli_tuning.sh --check-only

---

Performance Characteristics

Operation	Duration	Notes
Prerequisites check	~5s	Validates 10+ conditions
Job submission	~1-2s	gRPC call to API Gateway
Status polling	~5s/poll	Configurable via POLL_INTERVAL
Full workflow (5 trials)	~5-10min	Depends on model complexity
Quick test	~10s	Start job only

---

Security Considerations

1. JWT Token Masking: Tokens displayed as first20...last10 characters
2. Token Storage: ~/.foxhunt/jwt_token (chmod 600 recommended)
3. Cleanup on Exit: Jobs stopped automatically on script failure
4. No Hardcoded Credentials: All auth via TLI login flow
5. HTTPS Support: Change API_GATEWAY_URL for production

---

Future Enhancements

Planned Features

1. Real-time Streaming: Replace polling with server-side streaming gRPC
2. Multi-job Testing: Test multiple concurrent tuning jobs
3. Performance Metrics: Track latency, throughput, success rates
4. GPU Testing: Add --gpu flag validation
5. Data Validation: Verify parquet file structure before submission

Configuration Improvements

1. Custom Metrics: Support custom objective functions
2. Pruning Strategies: Test Optuna pruning algorithms
3. Multi-model Tests: Iterate over all supported models
4. Resource Limits: Memory/CPU constraints testing

---

Related Documentation

• TLI Tune Command: /home/jgrusewski/Work/foxhunt/tli/src/commands/tune.rs
• API Gateway Proxy: /home/jgrusewski/Work/foxhunt/services/api_gateway/src/proxy_handlers.rs
• ML Training Service: /home/jgrusewski/Work/foxhunt/services/ml_training_service/
• CLAUDE.md: Architecture and development guide
• TESTING_PLAN.md: ML testing strategy

---

Version History

1.0.0 (2025-10-13)

• Initial release
• Full workflow testing (start → poll → results)
• Three operating modes (full, quick, check-only)
• Comprehensive error handling
• Auto-config generation
• JWT token masking
• Real-time progress tracking
• Export best parameters to YAML

---

License

Part of the Foxhunt HFT Trading System Copyright (c) 2025