foxhunt/docs/WAVE77_AGENT4_ML_CLI_FIX.md

WAVE 77 AGENT 4: ML Training Service CLI Interface Fix

Agent: Wave 77 Agent 4 Date: 2025-10-03 Mission: Update deployment scripts to use correct CLI interface (serve subcommand)

Problem Statement

Wave 76 Agent 8 introduced a new CLI structure for ml_training_service that requires the serve subcommand to start the service. However, the deployment scripts were still using the old command format without the subcommand, causing service startup failures.

Error:

# Old command (broken):
./target/release/ml_training_service &> logs/ml_training.log &

# Required command:
./target/release/ml_training_service serve &> logs/ml_training.log &

Changes Made

1. Updated start_all_services.sh

File: /home/jgrusewski/Work/foxhunt/start_all_services.sh

Before (line 47):

./target/release/ml_training_service &> logs/ml_training.log &

After (line 47):

./target/release/ml_training_service serve &> logs/ml_training.log &

Impact: Service will now start correctly with the new CLI structure.

2. Updated create_systemd_services.sh

File: /home/jgrusewski/Work/foxhunt/deployment/create_systemd_services.sh

Added logic (lines 351-355) to conditionally append serve subcommand for ml_training_service:

# Determine if service needs subcommand
local exec_command="$DATA_DIR/bin/$binary_name"
if [[ "$binary_name" == "ml_training_service" ]]; then
    exec_command="$DATA_DIR/bin/$binary_name serve"
fi

Before (ExecStart):

ExecStart=/opt/foxhunt/bin/ml_training_service

After (ExecStart):

ExecStart=/opt/foxhunt/bin/ml_training_service serve

Impact: SystemD service files will be generated with correct command for ml_training_service.

ML Training Service CLI Interface

Available Commands

ML Training Service for Foxhunt HFT Trading System

Usage: ml_training_service <COMMAND>

Commands:
  serve     Start the ML training service
  health    Health check
  database  Database operations
  config    Configuration validation
  help      Print this message or the help of the given subcommand(s)

Options:
  -h, --help  Print help

Serve Subcommand Options

Start the ML training service

Usage: ml_training_service serve [OPTIONS]

Options:
  -c, --config <CONFIG>  Configuration file path
  -p, --port <PORT>      Override server port
      --dev              Enable development mode with debug logging
  -h, --help             Print help

Environment Variable Propagation

Confirmed: Environment variables still propagate correctly through the updated command:

# Environment loading (lines 6-9 in start_all_services.sh)
set -a
source .env
set +a

# Service startup with env vars
./target/release/ml_training_service serve &> logs/ml_training.log &

Environment variables available to ml_training_service:

• DATABASE_URL - PostgreSQL connection
• REDIS_URL - Redis connection
• GRPC_PORT - Override port (default: 50053)
• TLS_CA_PATH - TLS certificate authority path
• ENVIRONMENT - deployment environment
• All other .env variables

Verification

CLI Help Output

✅ Main CLI help shows all commands:

$ ./target/release/ml_training_service --help
ML Training Service for Foxhunt HFT Trading System

Usage: ml_training_service <COMMAND>
...

✅ Serve subcommand help works:

$ ./target/release/ml_training_service serve --help
Start the ML training service

Usage: ml_training_service serve [OPTIONS]
...

Deployment Scripts

✅ start_all_services.sh - Updated with serve subcommand ✅ create_systemd_services.sh - Conditional logic for ml_training_service ✅ Environment variable propagation verified ✅ No changes needed to other scripts (they don't invoke the binary directly)

Impact Analysis

Files Modified

1. /home/jgrusewski/Work/foxhunt/start_all_services.sh - Service startup script
2. /home/jgrusewski/Work/foxhunt/deployment/create_systemd_services.sh - SystemD generator

Files Checked (No Changes Needed)

• stop.sh - Uses pkill (process name only)
• health_check.sh - Uses health check endpoint
• quick_health_check.sh - Uses process detection
• Other deployment scripts - Don't invoke binary directly

Testing Recommendations

1. Development Testing

# Test service startup
./start_all_services.sh

# Check ml_training_service started correctly
ps aux | grep ml_training_service
tail -f logs/ml_training.log

# Test health check
./target/release/ml_training_service health --endpoint http://localhost:50053

2. SystemD Testing

# Generate SystemD service files
./deployment/create_systemd_services.sh --output-dir ./systemd

# Verify ml-training service file contains 'serve' subcommand
grep ExecStart ./systemd/foxhunt-ml-training.service
# Expected: ExecStart=/opt/foxhunt/bin/ml_training_service serve

3. Production Deployment

# Verify binary exists
ls -la target/release/ml_training_service

# Test serve command
./target/release/ml_training_service serve --help

# Deploy with updated scripts
./deployment/deploy_production.sh

Related Wave Fixes

This fix complements Wave 76 Agent 8's CLI modernization:

• Wave 76 Agent 8: Implemented CLI structure with subcommands
• Wave 77 Agent 4: Updated deployment scripts to use new CLI interface

Backward Compatibility

Breaking Change: The ml_training_service binary now REQUIRES a subcommand.

Migration Path:

1. ✅ Update start_all_services.sh (completed)
2. ✅ Update create_systemd_services.sh (completed)
3. 🔄 Update any custom deployment scripts to use ml_training_service serve
4. 🔄 Update documentation to reflect CLI change

Summary

Status: ✅ COMPLETE

Changes:

• Fixed service startup command in start_all_services.sh
• Updated SystemD service generator to append serve subcommand
• Verified environment variable propagation still works
• Confirmed CLI interface accepts serve subcommand

Testing Required:

• Development environment testing with start_all_services.sh
• SystemD service file generation and verification
• Production deployment with updated scripts

Result: ML training service will now start correctly with the new CLI interface in both development and production environments.