foxhunt/scripts/README_test_tli_tuning.md

# TLI Tuning Workflow Test Script

**Script**: `/home/jgrusewski/Work/foxhunt/scripts/test_tli_tuning.sh`
**Purpose**: End-to-end testing of the TLI hyperparameter tuning workflow
**Version**: 1.0.0
**Last Updated**: 2025-10-13

---

## Overview

This script provides comprehensive testing of the TLI hyperparameter tuning functionality, including:
- Prerequisite validation (services, authentication, data)
- Job submission and tracking
- Status polling with real-time progress
- Best parameter retrieval and export
- Graceful error handling and cleanup

## Features

### 1. Prerequisite Checks ✅
- TLI binary existence and location
- JWT token validation (with masking for security)
- API Gateway health (HTTP + gRPC)
- ML Training Service availability
- Config file creation (auto-generates if missing)
- Test data validation with size reporting
- Optional tool detection (grpcurl, jq)

### 2. Three Operating Modes

#### Full Workflow (Default)
```bash
./scripts/test_tli_tuning.sh
```
**Steps**:
1. Check prerequisites
2. Start tuning job (DQN, 5 trials)
3. Poll status every 5 seconds
4. Display real-time progress with timestamps
5. Retrieve best parameters when complete
6. Export results to YAML file

**Duration**: ~5-10 minutes (depending on trials)

#### Quick Test Mode
```bash
./scripts/test_tli_tuning.sh --quick
```
**Steps**:
1. Check prerequisites
2. Start tuning job
3. Display job ID and monitoring commands
4. Exit immediately (no polling)

**Duration**: ~10 seconds
**Use Case**: Verify job submission works without waiting

#### Check-Only Mode
```bash
./scripts/test_tli_tuning.sh --check-only
```
**Steps**:
1. Validate all prerequisites
2. Report status of services/files
3. Exit with detailed diagnostics

**Duration**: ~5 seconds
**Use Case**: Pre-flight checks before running tests

### 3. Customization Options

```bash
# Custom model type
./scripts/test_tli_tuning.sh --model PPO --trials 10

# Environment variable overrides
export POLL_INTERVAL=10         # Poll every 10 seconds
export MAX_WAIT_TIME=600        # Wait up to 10 minutes
export TLI_BIN=/custom/path/tli # Custom TLI location
./scripts/test_tli_tuning.sh
```

### 4. Error Handling

- **Automatic cleanup**: Stops jobs on script failure/interrupt
- **Exit traps**: SIGINT, SIGTERM handled gracefully
- **Detailed diagnostics**: Clear error messages with remediation steps
- **Progress tracking**: Job IDs saved to `~/.foxhunt/test_tuning_job.txt`

### 5. Output Features

- **Color-coded status**: ✅ Green (success), ❌ Red (error), ⚠️ Yellow (warning)
- **Real-time progress**: Timestamps, trial progress, elapsed time
- **Progress visualization**: Text-based progress bar
- **Duration tracking**: Total test duration reported
- **Masked credentials**: JWT tokens masked for security

---

## Prerequisites

### 1. Build TLI Binary
```bash
cargo build --release -p tli
# Binary: target/release/tli
```

### 2. Authenticate with TLI
```bash
./target/release/tli auth login --username <user> --password <pass>
# Creates: ~/.foxhunt/jwt_token
```

### 3. Start Infrastructure Services
```bash
docker-compose up -d
# Services: postgres, redis, vault, influxdb
```

### 4. Start API Gateway
```bash
cargo run --release -p api_gateway
# Ports: 50051 (gRPC), 8080 (HTTP health)
```

### 5. Start ML Training Service
```bash
cargo run --release -p ml_training_service
# Ports: 50054 (gRPC), 8095 (HTTP health)
```

### 6. Verify Health
```bash
./scripts/comprehensive_health_check.sh
# Should show all services healthy
```

---

## Usage Examples

### Example 1: Basic Full Workflow Test
```bash
$ ./scripts/test_tli_tuning.sh

================================================================================
  Prerequisite Checks
================================================================================

[STEP] Checking TLI binary...
✓ TLI binary found
ℹ Location: /home/jgrusewski/Work/foxhunt/target/release/tli

[STEP] Checking JWT token...
✓ JWT token found
ℹ Token (masked): eyJhbGciOiJIUzI1NiIs...WXZ6aGJHVnU=

[STEP] Checking API Gateway health...
✓ API Gateway is healthy

[STEP] Checking ML Training Service health...
✓ ML Training Service is healthy

[STEP] Checking tuning config file...
✓ Config file found
ℹ Path: /home/jgrusewski/Work/foxhunt/test_data/tuning_config.yaml

[STEP] Checking test data...
✓ Test data found
ℹ Path: /home/jgrusewski/Work/foxhunt/test_data/btcusdt_sample_100.parquet
ℹ Size: 1.2M

✓ All prerequisites satisfied

================================================================================
  Starting Tuning Job
================================================================================

[STEP] Submitting tuning job to TLI...
ℹ Command: ./target/release/tli tune start --model DQN --trials 5 --config test_data/tuning_config.yaml

🚀 Starting hyperparameter tuning job...
   Model: DQN
   Trials: 5
   Config: test_data/tuning_config.yaml
   GPU: ❌ Disabled

✅ Tuning job started successfully!
   Job ID: 550e8400-e29b-41d4-a716-446655440000
   Saved to ~/.foxhunt/tuning_jobs.json

================================================================================
  Polling Job Status
================================================================================

ℹ Polling every 5s (max wait: 5m)

[14:30:05] Status: RUNNING | Progress: 0/5 (0.0%) | Elapsed: 0s
[14:30:10] Status: RUNNING | Progress: 1/5 (20.0%) | Elapsed: 5s
[14:30:15] Status: RUNNING | Progress: 2/5 (40.0%) | Elapsed: 10s
[14:30:20] Status: RUNNING | Progress: 3/5 (60.0%) | Elapsed: 15s
[14:30:25] Status: RUNNING | Progress: 4/5 (80.0%) | Elapsed: 20s
[14:30:30] Status: RUNNING | Progress: 5/5 (100.0%) | Elapsed: 25s

✅ Job completed successfully!

📊 Tuning Job Status
   Status: TUNING_COMPLETED
   Progress: 5/5 trials (100.0%)
   [██████████████████████████████████████████████████] 100.0%

🏆 Best Results So Far
   Sharpe Ratio: 2.3456
   Elapsed Time: 30 seconds

================================================================================
  Retrieving Best Parameters
================================================================================

[STEP] Fetching best hyperparameters...

🏆 Best Performance Metrics
   sharpe_ratio: 2.3456
   total_return: 15.6789
   max_drawdown: 8.4321

📋 Best Hyperparameters
┌──────────────────┬──────────┬───────────────┐
│ Parameter        │ Value    │ Type          │
├──────────────────┼──────────┼───────────────┤
│ learning_rate    │ 0.000512 │ Learning Rate │
│ batch_size       │ 64.000000│ Integer       │
│ gamma            │ 0.976543 │ Float         │
│ epsilon_decay    │ 0.995678 │ Float         │
└──────────────────┴──────────┴───────────────┘

[STEP] Exporting best parameters to file...
✅ Parameters exported to: best_params_550e8400.yaml

💡 Use these parameters in your training configuration.

================================================================================
  Test Completed Successfully
================================================================================

✅ Full workflow executed without errors
ℹ Total duration: 35s
```

### Example 2: Quick Test (No Waiting)
```bash
$ ./scripts/test_tli_tuning.sh --quick

================================================================================
  Quick TLI Tuning Test (Start Only)
================================================================================

[Prerequisites checks...]
✅ All prerequisites satisfied

================================================================================
  Starting Tuning Job
================================================================================

✅ Tuning job started successfully!
   Job ID: 550e8400-e29b-41d4-a716-446655440000

✅ Job started successfully
ℹ Monitor with: ./target/release/tli tune status --job-id 550e8400-e29b-41d4-a716-446655440000
ℹ Get results: ./target/release/tli tune best --job-id 550e8400-e29b-41d4-a716-446655440000
ℹ Stop job: ./target/release/tli tune stop --job-id 550e8400-e29b-41d4-a716-446655440000
ℹ Total test duration: 8s
```

### Example 3: Custom Model and Trials
```bash
$ ./scripts/test_tli_tuning.sh --model MAMBA_2 --trials 10

[Prerequisites checks...]
✅ All prerequisites satisfied

🚀 Starting hyperparameter tuning job...
   Model: MAMBA_2
   Trials: 10
   Config: test_data/tuning_config.yaml
   GPU: ❌ Disabled

[Polling progress 0-100%...]
```

### Example 4: Check Prerequisites Only
```bash
$ ./scripts/test_tli_tuning.sh --check-only

================================================================================
  Prerequisite Checks
================================================================================

[STEP] Checking TLI binary...
✓ TLI binary found
ℹ Location: /home/jgrusewski/Work/foxhunt/target/release/tli

[STEP] Checking JWT token...
✗ JWT token not found at: /home/jgrusewski/.foxhunt/jwt_token
ℹ Authenticate with: ./target/release/tli auth login

[STEP] Checking API Gateway health...
✗ API Gateway not responding at http://localhost:8080/health
ℹ Start services with: docker-compose up -d

[STEP] Checking ML Training Service health...
✓ ML Training Service is healthy

✗ Prerequisites check FAILED
ℹ Please resolve issues above and retry
```

---

## Configuration

### Environment Variables

| Variable | Default | Description |
|----------|---------|-------------|
| `TLI_BIN` | `target/release/tli` | Path to TLI binary |
| `JWT_TOKEN_FILE` | `~/.foxhunt/jwt_token` | JWT token location |
| `POLL_INTERVAL` | `5` | Status polling interval (seconds) |
| `MAX_WAIT_TIME` | `300` | Maximum wait time (seconds) |

### Generated Files

| File | Location | Purpose |
|------|----------|---------|
| JWT Token | `~/.foxhunt/jwt_token` | Authentication |
| Tuning Jobs | `~/.foxhunt/tuning_jobs.json` | Job tracking |
| Test Job ID | `~/.foxhunt/test_tuning_job.txt` | Current test job |
| Minimal Config | `test_data/tuning_config.yaml` | Auto-generated config |
| Best Params | `best_params_<job_id>.yaml` | Exported parameters |

### Minimal Config Structure

The script auto-generates a minimal config if none exists:

```yaml
# test_data/tuning_config.yaml
tuning:
  study_name: "test_study"
  storage: "sqlite:///optuna_test.db"
  direction: "maximize"  # Maximize Sharpe ratio

  search_space:
    learning_rate:
      type: "float"
      low: 0.0001
      high: 0.01
      log: true

    batch_size:
      type: "int"
      low: 32
      high: 128
      step: 32

    gamma:
      type: "float"
      low: 0.90
      high: 0.99

    epsilon_decay:
      type: "float"
      low: 0.990
      high: 0.999

  training:
    epochs: 10
    validation_split: 0.2
    early_stopping_patience: 3

  metrics:
    - "sharpe_ratio"
    - "total_return"
    - "max_drawdown"
    - "win_rate"
```

---

## Troubleshooting

### Issue: TLI Binary Not Found

**Error**:
```
✗ TLI binary not found at: target/release/tli
ℹ Build with: cargo build --release -p tli
```

**Solution**:
```bash
cargo build --release -p tli
```

---

### Issue: JWT Token Missing

**Error**:
```
✗ JWT token not found at: ~/.foxhunt/jwt_token
ℹ Authenticate with: ./target/release/tli auth login
```

**Solution**:
```bash
./target/release/tli auth login --username admin --password admin123
```

**Note**: Default dev credentials from `docker-compose.yml`

---

### Issue: API Gateway Not Running

**Error**:
```
✗ API Gateway not responding at http://localhost:8080/health
ℹ Start services with: docker-compose up -d
```

**Solution**:
```bash
# Start infrastructure
docker-compose up -d

# Start API Gateway
cargo run --release -p api_gateway
```

---

### Issue: ML Training Service Unavailable

**Error**:
```
✗ ML Training Service not responding at http://localhost:8095/health
```

**Solution**:
```bash
# Check service status
curl http://localhost:8095/health

# Start if not running
cargo run --release -p ml_training_service

# Check logs
docker-compose logs ml_training_service
```

---

### Issue: Job Times Out

**Symptom**: Job exceeds MAX_WAIT_TIME (300s default)

**Solution**:
```bash
# Increase max wait time
export MAX_WAIT_TIME=600  # 10 minutes
./scripts/test_tli_tuning.sh

# Or reduce trials for faster completion
./scripts/test_tli_tuning.sh --trials 3
```

---

### Issue: Config File Not Found

**Symptom**: Warning about missing config file

**Behavior**: Script auto-creates minimal config at `test_data/tuning_config.yaml`

**Verification**:
```bash
cat test_data/tuning_config.yaml
```

---

## Integration with CI/CD

### GitHub Actions Example
```yaml
name: Test TLI Tuning
on: [push, pull_request]

jobs:
  test-tli-tuning:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2

      - name: Start infrastructure
        run: docker-compose up -d

      - name: Build TLI
        run: cargo build --release -p tli

      - name: Start services
        run: |
          cargo run --release -p api_gateway &
          cargo run --release -p ml_training_service &
          sleep 10

      - name: Authenticate TLI
        run: |
          ./target/release/tli auth login \
            --username admin \
            --password ${{ secrets.TLI_PASSWORD }}

      - name: Run quick test
        run: ./scripts/test_tli_tuning.sh --quick

      - name: Check prerequisites
        run: ./scripts/test_tli_tuning.sh --check-only
```

---

## Performance Characteristics

| Operation | Duration | Notes |
|-----------|----------|-------|
| Prerequisites check | ~5s | Validates 10+ conditions |
| Job submission | ~1-2s | gRPC call to API Gateway |
| Status polling | ~5s/poll | Configurable via `POLL_INTERVAL` |
| Full workflow (5 trials) | ~5-10min | Depends on model complexity |
| Quick test | ~10s | Start job only |

---

## Security Considerations

1. **JWT Token Masking**: Tokens displayed as `first20...last10` characters
2. **Token Storage**: `~/.foxhunt/jwt_token` (chmod 600 recommended)
3. **Cleanup on Exit**: Jobs stopped automatically on script failure
4. **No Hardcoded Credentials**: All auth via TLI login flow
5. **HTTPS Support**: Change `API_GATEWAY_URL` for production

---

## Future Enhancements

### Planned Features
1. **Real-time Streaming**: Replace polling with server-side streaming gRPC
2. **Multi-job Testing**: Test multiple concurrent tuning jobs
3. **Performance Metrics**: Track latency, throughput, success rates
4. **GPU Testing**: Add `--gpu` flag validation
5. **Data Validation**: Verify parquet file structure before submission

### Configuration Improvements
1. **Custom Metrics**: Support custom objective functions
2. **Pruning Strategies**: Test Optuna pruning algorithms
3. **Multi-model Tests**: Iterate over all supported models
4. **Resource Limits**: Memory/CPU constraints testing

---

## Related Documentation

- **TLI Tune Command**: `/home/jgrusewski/Work/foxhunt/tli/src/commands/tune.rs`
- **API Gateway Proxy**: `/home/jgrusewski/Work/foxhunt/services/api_gateway/src/proxy_handlers.rs`
- **ML Training Service**: `/home/jgrusewski/Work/foxhunt/services/ml_training_service/`
- **CLAUDE.md**: Architecture and development guide
- **TESTING_PLAN.md**: ML testing strategy

---

## Version History

### 1.0.0 (2025-10-13)
- Initial release
- Full workflow testing (start → poll → results)
- Three operating modes (full, quick, check-only)
- Comprehensive error handling
- Auto-config generation
- JWT token masking
- Real-time progress tracking
- Export best parameters to YAML

---

## License

Part of the Foxhunt HFT Trading System
Copyright (c) 2025