foxhunt/docs/TLI_OPERATIONS_MANUAL.md

# TLI Operations Manual
**Foxhunt Trading System - Terminal Line Interface**

Version: 1.0
Last Updated: 2025-01-23
Document Classification: Production Operations

---

## Table of Contents

1. [Overview](#overview)
2. [System Prerequisites](#system-prerequisites)
3. [Service Startup Procedures](#service-startup-procedures)
4. [Configuration Management](#configuration-management)
5. [Daily Operations](#daily-operations)
6. [Monitoring and Health Checks](#monitoring-and-health-checks)
7. [Troubleshooting](#troubleshooting)
8. [Performance Tuning](#performance-tuning)
9. [Emergency Procedures](#emergency-procedures)
10. [Maintenance Procedures](#maintenance-procedures)

---

## Overview

The TLI (Terminal Line Interface) is the primary gRPC-based client interface for the Foxhunt HFT Trading System. It provides secure, high-performance access to:

- **Trading Operations**: Order management, execution, portfolio tracking
- **Risk Management**: VaR calculations, position limits, compliance monitoring
- **Market Data**: Real-time streaming, historical data access
- **Backtesting**: Strategy testing and performance analysis
- **System Monitoring**: Metrics, latency tracking, health status
- **Configuration**: Dynamic parameter updates, system configuration

### Architecture Overview

```
TLI Client Suite
├── Connection Manager (pooling, health checks, reconnection)
├── Event Stream Manager (real-time data streaming)
├── Trading Client (unified trading, risk, monitoring, config)
└── Backtesting Client (strategy testing, performance analysis)
```

---

## System Prerequisites

### Hardware Requirements

- **CPU**: Minimum 8 cores, recommended 16+ cores for production
- **Memory**: Minimum 16GB RAM, recommended 32GB+ for high-frequency operations
- **Network**: Low-latency network connection (< 1ms to trading venues)
- **Storage**: SSD storage for database and logs

### Software Dependencies

- **Rust**: Version 1.75+ (for compilation)
- **gRPC**: Included in dependencies
- **TLS Certificates**: Required for production security
- **Database**: PostgreSQL for audit logs, Redis for caching

### Network Configuration

- **Port 50051**: Trading Service (default)
- **Port 50052**: Backtesting Service (default)
- **Port 443**: HTTPS/TLS for secure communications
- **Firewall**: Configure for service discovery and health checks

---

## Service Startup Procedures

### 1. Pre-Startup Checklist

```bash
# Verify certificate files exist and are valid
ls -la /etc/foxhunt/tls/
# Expected files: server.crt, server.key, ca.crt

# Check database connectivity
psql -h localhost -U foxhunt_user -d foxhunt_db -c "SELECT 1;"

# Verify Redis connectivity
redis-cli ping

# Check system resources
free -h
df -h
```

### 2. Environment Setup

```bash
# Set environment variables
export FOXHUNT_ENV=production
export RUST_LOG=info
export FOXHUNT_CONFIG_PATH=/etc/foxhunt/config.toml

# Source environment configuration
source /etc/foxhunt/environment
```

### 3. Service Startup Sequence

#### Option A: Systemd (Recommended for Production)

```bash
# Start core services first
sudo systemctl start foxhunt-database
sudo systemctl start foxhunt-redis

# Start trading services
sudo systemctl start foxhunt-trading-service
sudo systemctl start foxhunt-backtesting-service

# Verify services are running
sudo systemctl status foxhunt-trading-service
sudo systemctl status foxhunt-backtesting-service
```

#### Option B: Docker Deployment

```bash
# Start via docker-compose
cd /opt/foxhunt
docker-compose up -d

# Verify containers are healthy
docker-compose ps
docker-compose logs -f
```

#### Option C: Manual Startup (Development)

```bash
# Start trading service
cd /opt/foxhunt
RUST_LOG=info ./target/release/trading-service &

# Start backtesting service
RUST_LOG=info ./target/release/backtesting-service &

# Verify processes
ps aux | grep foxhunt
```

### 4. Post-Startup Verification

```bash
# Test gRPC connectivity
grpcurl -insecure localhost:50051 list

# Check health endpoints
curl -k https://localhost:50051/health
curl -k https://localhost:50052/health

# Verify TLI client connectivity
cargo run --bin tli-client -- --command ping
```

---

## Configuration Management

### Configuration Files

| File | Purpose | Location |
|------|---------|----------|
| `config.toml` | Main service configuration | `/etc/foxhunt/config.toml` |
| `security.toml` | Authentication and TLS settings | `/etc/foxhunt/security.toml` |
| `logging.toml` | Logging configuration | `/etc/foxhunt/logging.toml` |
| `environment` | Environment variables | `/etc/foxhunt/environment` |

### Main Configuration Structure

```toml
[trading_service]
host = "0.0.0.0"
port = 50051
max_connections = 1000
timeout_seconds = 30

[backtesting_service]
host = "0.0.0.0"
port = 50052
max_connections = 100
timeout_seconds = 300

[database]
url = "postgresql://user:pass@localhost/foxhunt_db"
max_connections = 50
timeout_seconds = 10

[redis]
url = "redis://localhost:6379"
pool_size = 20
timeout_seconds = 5

[security]
tls_cert_path = "/etc/foxhunt/tls/server.crt"
tls_key_path = "/etc/foxhunt/tls/server.key"
ca_cert_path = "/etc/foxhunt/tls/ca.crt"
require_client_cert = true

[rate_limiting]
authenticated_rpm = 1000
api_key_rpm = 5000
trading_burst = 100
window_seconds = 60

[audit]
log_auth_attempts = true
log_trading_operations = true
retention_days = 2555
encrypt_logs = true
```

### Dynamic Configuration Updates

```bash
# Update configuration via TLI client
cargo run --bin tli-client -- --command config-update \
  --key "rate_limiting.trading_burst" \
  --value "150"

# Reload configuration without restart
kill -HUP $(pidof trading-service)

# Verify configuration changes
cargo run --bin tli-client -- --command config-get \
  --key "rate_limiting.trading_burst"
```

### Configuration Backup and Restore

```bash
# Backup current configuration
cp /etc/foxhunt/config.toml /etc/foxhunt/config.toml.backup.$(date +%Y%m%d)

# Restore configuration
cp /etc/foxhunt/config.toml.backup.20250123 /etc/foxhunt/config.toml
sudo systemctl reload foxhunt-trading-service
```

---

## Daily Operations

### Morning Startup Checklist

1. **System Health Check**
   ```bash
   # Check system status
   cargo run --bin tli-client -- --command system-status

   # Verify all services are healthy
   sudo systemctl status foxhunt-*
   ```

2. **Market Data Validation**
   ```bash
   # Test market data connectivity
   cargo run --bin tli-client -- --command market-data-test

   # Verify real-time feeds
   cargo run --bin tli-client -- --command stream-test --symbols AAPL,GOOGL
   ```

3. **Risk System Verification**
   ```bash
   # Check risk limits
   cargo run --bin tli-client -- --command risk-limits-check

   # Verify VaR calculations
   cargo run --bin tli-client -- --command var-test
   ```

### End-of-Day Procedures

1. **Position Reconciliation**
   ```bash
   # Generate position report
   cargo run --bin tli-client -- --command positions-report \
     --format json > /var/log/foxhunt/positions-$(date +%Y%m%d).json
   ```

2. **Performance Summary**
   ```bash
   # Generate daily performance report
   cargo run --bin tli-client -- --command performance-summary \
     --date $(date +%Y-%m-%d)
   ```

3. **Log Rotation**
   ```bash
   # Rotate application logs
   logrotate /etc/logrotate.d/foxhunt

   # Archive audit logs
   /opt/foxhunt/scripts/archive-audit-logs.sh
   ```

---

## Monitoring and Health Checks

### Health Check Endpoints

| Service | Endpoint | Expected Response |
|---------|----------|-------------------|
| Trading Service | `localhost:50051/health` | `HTTP 200 OK` |
| Backtesting Service | `localhost:50052/health` | `HTTP 200 OK` |
| TLI Client | `tli-client --command ping` | `PONG` response |

### Key Metrics to Monitor

1. **Performance Metrics**
   - Order submission latency (target: < 50μs)
   - Market data processing latency (target: < 10μs)
   - gRPC request throughput
   - Memory usage and garbage collection

2. **Business Metrics**
   - Active trading sessions
   - Orders per second
   - Portfolio value-at-risk
   - Risk limit violations

3. **System Metrics**
   - CPU utilization
   - Memory usage
   - Network throughput
   - Disk I/O

### Monitoring Commands

```bash
# Real-time metrics dashboard
cargo run --bin tli-client -- --command metrics-dashboard

# Latency monitoring
cargo run --bin tli-client -- --command latency-monitor \
  --interval 5s --duration 1h

# Throughput monitoring
cargo run --bin tli-client -- --command throughput-monitor \
  --service trading --operation submit_order
```

### Alerting Thresholds

| Metric | Warning | Critical |
|--------|---------|----------|
| Order Latency | > 100μs | > 500μs |
| CPU Usage | > 70% | > 90% |
| Memory Usage | > 80% | > 95% |
| Error Rate | > 1% | > 5% |
| Risk Violations | Any | Multiple |

---

## Troubleshooting

### Common Issues and Solutions

#### Issue: "Connection Refused" Error

**Symptoms:**
```
Error: Transport error: Connection refused (os error 111)
```

**Diagnosis:**
```bash
# Check if service is running
ps aux | grep trading-service

# Check port binding
netstat -tlnp | grep 50051

# Check firewall
sudo iptables -L | grep 50051
```

**Solution:**
```bash
# Restart service
sudo systemctl restart foxhunt-trading-service

# Check logs for startup errors
journalctl -u foxhunt-trading-service -f
```

#### Issue: TLS Certificate Errors

**Symptoms:**
```
Error: Certificate validation failed: certificate has expired
```

**Diagnosis:**
```bash
# Check certificate expiration
openssl x509 -in /etc/foxhunt/tls/server.crt -text -noout | grep "Not After"

# Validate certificate chain
openssl verify -CAfile /etc/foxhunt/tls/ca.crt /etc/foxhunt/tls/server.crt
```

**Solution:**
```bash
# Generate new certificates
/opt/foxhunt/scripts/generate-certificates.sh

# Restart services
sudo systemctl restart foxhunt-trading-service
```

#### Issue: High Latency

**Symptoms:**
- Order submission taking > 100μs
- Market data delays

**Diagnosis:**
```bash
# Check system load
top
iostat 1

# Check network latency
ping trading-venue.com

# Check CPU affinity
taskset -p $(pidof trading-service)
```

**Solution:**
```bash
# Set CPU affinity for performance
sudo taskset -c 0,1 $(pidof trading-service)

# Increase process priority
sudo renice -10 $(pidof trading-service)

# Check for other processes using CPU
ps aux --sort=-%cpu | head -20
```

#### Issue: Authentication Failures

**Symptoms:**
```
Error: Access denied: insufficient permissions for trade:execute
```

**Diagnosis:**
```bash
# Check user permissions
cargo run --bin tli-client -- --command check-permissions \
  --user trader_user_id --permission trade:execute

# Check API key status
cargo run --bin tli-client -- --command api-key-status \
  --key-id abc123
```

**Solution:**
```bash
# Update user permissions
cargo run --bin tli-client -- --command grant-permission \
  --user trader_user_id --permission trade:execute

# Regenerate API key if expired
cargo run --bin tli-client -- --command create-api-key \
  --user trader_user_id --name "Trading Bot" \
  --permissions trade:execute,order:place
```

### Log Analysis

#### Application Logs
```bash
# View real-time logs
tail -f /var/log/foxhunt/trading-service.log

# Search for errors
grep -i error /var/log/foxhunt/trading-service.log | tail -50

# Filter by timestamp
grep "2025-01-23 14:" /var/log/foxhunt/trading-service.log
```

#### Audit Logs
```bash
# View authentication attempts
grep "auth_attempt" /var/log/foxhunt/audit.log | tail -20

# View trading operations
grep "trade_operation" /var/log/foxhunt/audit.log | tail -20

# Search for specific user activity
grep "user_id:trader_001" /var/log/foxhunt/audit.log
```

---

## Performance Tuning

### Operating System Tuning

```bash
# Increase file descriptor limits
echo "* soft nofile 65536" >> /etc/security/limits.conf
echo "* hard nofile 65536" >> /etc/security/limits.conf

# TCP tuning for low latency
echo 'net.core.rmem_max = 134217728' >> /etc/sysctl.conf
echo 'net.core.wmem_max = 134217728' >> /etc/sysctl.conf
echo 'net.ipv4.tcp_rmem = 4096 87380 134217728' >> /etc/sysctl.conf
echo 'net.ipv4.tcp_wmem = 4096 65536 134217728' >> /etc/sysctl.conf
sysctl -p
```

### CPU Affinity and Process Priority

```bash
# Set CPU affinity for trading service (cores 0-3)
taskset -c 0-3 $(pidof trading-service)

# Set high priority
renice -15 $(pidof trading-service)

# Isolate CPUs for trading (add to kernel parameters)
# isolcpus=0,1,2,3 nohz_full=0,1,2,3 rcu_nocbs=0,1,2,3
```

### Memory Configuration

```bash
# Disable swap for predictable performance
swapoff -a

# Configure huge pages
echo 1024 > /proc/sys/vm/nr_hugepages

# Set memory overcommit
echo 1 > /proc/sys/vm/overcommit_memory
```

### Service-Specific Tuning

#### Trading Service Configuration
```toml
[performance]
connection_pool_size = 100
worker_threads = 16
max_blocking_threads = 512
thread_stack_size = 2097152
enable_thread_pinning = true
enable_numa_binding = true

[latency_optimization]
disable_nagle = true
tcp_nodelay = true
socket_recv_buffer = 1048576
socket_send_buffer = 1048576
```

#### Database Tuning
```sql
-- PostgreSQL tuning for trading workload
ALTER SYSTEM SET shared_buffers = '8GB';
ALTER SYSTEM SET effective_cache_size = '24GB';
ALTER SYSTEM SET maintenance_work_mem = '2GB';
ALTER SYSTEM SET checkpoint_completion_target = 0.9;
ALTER SYSTEM SET wal_buffers = '16MB';
ALTER SYSTEM SET default_statistics_target = 100;
SELECT pg_reload_conf();
```

### Monitoring Performance

```bash
# Monitor latency continuously
cargo run --bin tli-client -- --command latency-monitor \
  --output /var/log/foxhunt/latency.log

# Performance profiling
perf record -g ./target/release/trading-service
perf report

# Memory profiling
valgrind --tool=massif ./target/release/trading-service
```

---

## Emergency Procedures

### Emergency Stop Procedures

#### Immediate Market Stop
```bash
# Stop all trading immediately
cargo run --bin tli-client -- --command emergency-stop \
  --type full_shutdown --reason "Market emergency" --confirm

# Cancel all open orders
cargo run --bin tli-client -- --command emergency-stop \
  --type cancel_orders --confirm

# Close all positions
cargo run --bin tli-client -- --command emergency-stop \
  --type close_positions --confirm
```

#### Service Emergency Shutdown
```bash
# Graceful shutdown with position preservation
sudo systemctl stop foxhunt-trading-service

# Force shutdown if graceful fails
sudo kill -9 $(pidof trading-service)

# Emergency database backup
pg_dump foxhunt_db > /backup/emergency-$(date +%Y%m%d-%H%M%S).sql
```

### Disaster Recovery

#### Data Backup Verification
```bash
# Verify database backup
pg_restore --list /backup/latest-backup.sql

# Test configuration backup
tar -tzf /backup/config-backup.tar.gz

# Verify audit log integrity
/opt/foxhunt/scripts/verify-audit-logs.sh
```

#### Service Recovery
```bash
# Restore from backup
systemctl stop foxhunt-trading-service
pg_restore -d foxhunt_db /backup/latest-backup.sql
tar -xzf /backup/config-backup.tar.gz -C /etc/foxhunt/
systemctl start foxhunt-trading-service

# Verify recovery
cargo run --bin tli-client -- --command system-status
```

### Incident Response

#### Security Incident
1. **Immediate Actions**
   ```bash
   # Revoke all API keys
   cargo run --bin tli-client -- --command revoke-all-api-keys

   # Force logout all sessions
   cargo run --bin tli-client -- --command logout-all-sessions

   # Block suspicious IPs
   iptables -A INPUT -s <suspicious_ip> -j DROP
   ```

2. **Evidence Collection**
   ```bash
   # Backup audit logs
   cp /var/log/foxhunt/audit.log /secure/incident-$(date +%Y%m%d)/

   # Capture system state
   ps aux > /secure/incident-$(date +%Y%m%d)/processes.txt
   netstat -tulnp > /secure/incident-$(date +%Y%m%d)/network.txt
   ```

3. **Recovery**
   ```bash
   # Restore from clean backup
   # Regenerate all certificates
   # Reset all passwords and API keys
   # Review and update security policies
   ```

---

## Maintenance Procedures

### Scheduled Maintenance

#### Weekly Maintenance
- **Certificate rotation check**
- **Log file rotation and archival**
- **Performance metrics analysis**
- **Security audit log review**
- **Database maintenance (VACUUM, REINDEX)**

#### Monthly Maintenance
- **Full system backup verification**
- **Disaster recovery test**
- **Security penetration testing**
- **Performance benchmark comparison**
- **Dependencies update review**

#### Quarterly Maintenance
- **Full security audit**
- **Certificate renewal**
- **Hardware performance review**
- **Disaster recovery full test**
- **Compliance documentation review**

### Update Procedures

#### Binary Updates
```bash
# Create backup
cp /opt/foxhunt/target/release/trading-service \
   /backup/trading-service.$(date +%Y%m%d)

# Deploy new binary
sudo systemctl stop foxhunt-trading-service
cp /staging/trading-service /opt/foxhunt/target/release/
sudo systemctl start foxhunt-trading-service

# Verify deployment
cargo run --bin tli-client -- --command version
cargo run --bin tli-client -- --command health-check
```

#### Configuration Updates
```bash
# Backup current config
cp /etc/foxhunt/config.toml /backup/config.toml.$(date +%Y%m%d)

# Apply new configuration
cp /staging/config.toml /etc/foxhunt/
sudo systemctl reload foxhunt-trading-service

# Verify configuration
cargo run --bin tli-client -- --command config-validate
```

### Database Maintenance

#### Daily Maintenance
```sql
-- Analyze tables for query optimization
ANALYZE;

-- Check for long-running queries
SELECT * FROM pg_stat_activity WHERE state = 'active' AND now() - query_start > interval '1 minute';
```

#### Weekly Maintenance
```sql
-- Vacuum to reclaim space
VACUUM (ANALYZE, VERBOSE);

-- Reindex for performance
REINDEX DATABASE foxhunt_db;

-- Check database size
SELECT pg_size_pretty(pg_database_size('foxhunt_db'));
```

#### Monthly Maintenance
```bash
# Full backup
pg_dump foxhunt_db | gzip > /backup/monthly-$(date +%Y%m%d).sql.gz

# Backup verification
pg_restore --list /backup/monthly-$(date +%Y%m%d).sql.gz | head -20

# Archive old audit logs
/opt/foxhunt/scripts/archive-old-logs.sh --older-than 90days
```

---

## Contacts and Escalation

### Support Contacts

| Role | Contact | Phone | Email |
|------|---------|-------|--------|
| Primary On-Call | Trading Team | +1-555-0100 | trading-oncall@company.com |
| Secondary On-Call | Infrastructure Team | +1-555-0101 | infra-oncall@company.com |
| Database Admin | DBA Team | +1-555-0102 | dba@company.com |
| Security Team | Security Team | +1-555-0103 | security@company.com |

### Escalation Procedures

1. **Level 1**: Service degradation, non-critical issues
2. **Level 2**: Service outage, trading impact
3. **Level 3**: Security incident, data breach
4. **Level 4**: Regulatory incident, financial loss

### Emergency Contacts

- **Trading Floor**: +1-555-0200
- **Risk Management**: +1-555-0201
- **Compliance**: +1-555-0202
- **Executive Team**: +1-555-0203

---

*This operations manual is maintained by the Trading Infrastructure Team. For updates or corrections, please contact: trading-infrastructure@company.com*