foxhunt/docs/WAVE77_AGENT4_ML_CLI_FIX.md

# WAVE 77 AGENT 4: ML Training Service CLI Interface Fix

**Agent**: Wave 77 Agent 4
**Date**: 2025-10-03
**Mission**: Update deployment scripts to use correct CLI interface (serve subcommand)

## Problem Statement

Wave 76 Agent 8 introduced a new CLI structure for ml_training_service that requires the `serve` subcommand to start the service. However, the deployment scripts were still using the old command format without the subcommand, causing service startup failures.

**Error**:
```bash
# Old command (broken):
./target/release/ml_training_service &> logs/ml_training.log &

# Required command:
./target/release/ml_training_service serve &> logs/ml_training.log &
```

## Changes Made

### 1. Updated `start_all_services.sh`

**File**: `/home/jgrusewski/Work/foxhunt/start_all_services.sh`

**Before (line 47)**:
```bash
./target/release/ml_training_service &> logs/ml_training.log &
```

**After (line 47)**:
```bash
./target/release/ml_training_service serve &> logs/ml_training.log &
```

**Impact**: Service will now start correctly with the new CLI structure.

### 2. Updated `create_systemd_services.sh`

**File**: `/home/jgrusewski/Work/foxhunt/deployment/create_systemd_services.sh`

**Added logic (lines 351-355)** to conditionally append `serve` subcommand for ml_training_service:

```bash
# Determine if service needs subcommand
local exec_command="$DATA_DIR/bin/$binary_name"
if [[ "$binary_name" == "ml_training_service" ]]; then
    exec_command="$DATA_DIR/bin/$binary_name serve"
fi
```

**Before (ExecStart)**:
```ini
ExecStart=/opt/foxhunt/bin/ml_training_service
```

**After (ExecStart)**:
```ini
ExecStart=/opt/foxhunt/bin/ml_training_service serve
```

**Impact**: SystemD service files will be generated with correct command for ml_training_service.

## ML Training Service CLI Interface

### Available Commands

```
ML Training Service for Foxhunt HFT Trading System

Usage: ml_training_service <COMMAND>

Commands:
  serve     Start the ML training service
  health    Health check
  database  Database operations
  config    Configuration validation
  help      Print this message or the help of the given subcommand(s)

Options:
  -h, --help  Print help
```

### Serve Subcommand Options

```
Start the ML training service

Usage: ml_training_service serve [OPTIONS]

Options:
  -c, --config <CONFIG>  Configuration file path
  -p, --port <PORT>      Override server port
      --dev              Enable development mode with debug logging
  -h, --help             Print help
```

## Environment Variable Propagation

**Confirmed**: Environment variables still propagate correctly through the updated command:

```bash
# Environment loading (lines 6-9 in start_all_services.sh)
set -a
source .env
set +a

# Service startup with env vars
./target/release/ml_training_service serve &> logs/ml_training.log &
```

**Environment variables available to ml_training_service**:
- `DATABASE_URL` - PostgreSQL connection
- `REDIS_URL` - Redis connection
- `GRPC_PORT` - Override port (default: 50053)
- `TLS_CA_PATH` - TLS certificate authority path
- `ENVIRONMENT` - deployment environment
- All other `.env` variables

## Verification

### CLI Help Output

✅ Main CLI help shows all commands:
```bash
$ ./target/release/ml_training_service --help
ML Training Service for Foxhunt HFT Trading System

Usage: ml_training_service <COMMAND>
...
```

✅ Serve subcommand help works:
```bash
$ ./target/release/ml_training_service serve --help
Start the ML training service

Usage: ml_training_service serve [OPTIONS]
...
```

### Deployment Scripts

✅ `start_all_services.sh` - Updated with `serve` subcommand
✅ `create_systemd_services.sh` - Conditional logic for ml_training_service
✅ Environment variable propagation verified
✅ No changes needed to other scripts (they don't invoke the binary directly)

## Impact Analysis

### Files Modified
1. `/home/jgrusewski/Work/foxhunt/start_all_services.sh` - Service startup script
2. `/home/jgrusewski/Work/foxhunt/deployment/create_systemd_services.sh` - SystemD generator

### Files Checked (No Changes Needed)
- `stop.sh` - Uses pkill (process name only)
- `health_check.sh` - Uses health check endpoint
- `quick_health_check.sh` - Uses process detection
- Other deployment scripts - Don't invoke binary directly

## Testing Recommendations

### 1. Development Testing
```bash
# Test service startup
./start_all_services.sh

# Check ml_training_service started correctly
ps aux | grep ml_training_service
tail -f logs/ml_training.log

# Test health check
./target/release/ml_training_service health --endpoint http://localhost:50053
```

### 2. SystemD Testing
```bash
# Generate SystemD service files
./deployment/create_systemd_services.sh --output-dir ./systemd

# Verify ml-training service file contains 'serve' subcommand
grep ExecStart ./systemd/foxhunt-ml-training.service
# Expected: ExecStart=/opt/foxhunt/bin/ml_training_service serve
```

### 3. Production Deployment
```bash
# Verify binary exists
ls -la target/release/ml_training_service

# Test serve command
./target/release/ml_training_service serve --help

# Deploy with updated scripts
./deployment/deploy_production.sh
```

## Related Wave Fixes

This fix complements Wave 76 Agent 8's CLI modernization:
- **Wave 76 Agent 8**: Implemented CLI structure with subcommands
- **Wave 77 Agent 4**: Updated deployment scripts to use new CLI interface

## Backward Compatibility

**Breaking Change**: The ml_training_service binary now REQUIRES a subcommand.

**Migration Path**:
1. ✅ Update `start_all_services.sh` (completed)
2. ✅ Update `create_systemd_services.sh` (completed)
3. 🔄 Update any custom deployment scripts to use `ml_training_service serve`
4. 🔄 Update documentation to reflect CLI change

## Summary

**Status**: ✅ COMPLETE

**Changes**:
- Fixed service startup command in `start_all_services.sh`
- Updated SystemD service generator to append `serve` subcommand
- Verified environment variable propagation still works
- Confirmed CLI interface accepts `serve` subcommand

**Testing Required**:
- Development environment testing with `start_all_services.sh`
- SystemD service file generation and verification
- Production deployment with updated scripts

**Result**: ML training service will now start correctly with the new CLI interface in both development and production environments.