foxhunt/docs/TLI_OPERATIONS_MANUAL.md

TLI Operations Manual

Foxhunt Trading System - Terminal Line Interface

Version: 1.0 Last Updated: 2025-01-23 Document Classification: Production Operations

---

Table of Contents

1. Overview
2. System Prerequisites
3. Service Startup Procedures
4. Configuration Management
5. Daily Operations
6. Monitoring and Health Checks
7. Troubleshooting
8. Performance Tuning
9. Emergency Procedures
10. Maintenance Procedures

---

Overview

The TLI (Terminal Line Interface) is the primary gRPC-based client interface for the Foxhunt HFT Trading System. It provides secure, high-performance access to:

• Trading Operations: Order management, execution, portfolio tracking
• Risk Management: VaR calculations, position limits, compliance monitoring
• Market Data: Real-time streaming, historical data access
• Backtesting: Strategy testing and performance analysis
• System Monitoring: Metrics, latency tracking, health status
• Configuration: Dynamic parameter updates, system configuration

Architecture Overview

TLI Client Suite
├── Connection Manager (pooling, health checks, reconnection)
├── Event Stream Manager (real-time data streaming)
├── Trading Client (unified trading, risk, monitoring, config)
└── Backtesting Client (strategy testing, performance analysis)

---

System Prerequisites

Hardware Requirements

• CPU: Minimum 8 cores, recommended 16+ cores for production
• Memory: Minimum 16GB RAM, recommended 32GB+ for high-frequency operations
• Network: Low-latency network connection (< 1ms to trading venues)
• Storage: SSD storage for database and logs

Software Dependencies

• Rust: Version 1.75+ (for compilation)
• gRPC: Included in dependencies
• TLS Certificates: Required for production security
• Database: PostgreSQL for audit logs, Redis for caching

Network Configuration

• Port 50051: Trading Service (default)
• Port 50052: Backtesting Service (default)
• Port 443: HTTPS/TLS for secure communications
• Firewall: Configure for service discovery and health checks

---

Service Startup Procedures

1. Pre-Startup Checklist

# Verify certificate files exist and are valid
ls -la /etc/foxhunt/tls/
# Expected files: server.crt, server.key, ca.crt

# Check database connectivity
psql -h localhost -U foxhunt_user -d foxhunt_db -c "SELECT 1;"

# Verify Redis connectivity
redis-cli ping

# Check system resources
free -h
df -h

2. Environment Setup

# Set environment variables
export FOXHUNT_ENV=production
export RUST_LOG=info
export FOXHUNT_CONFIG_PATH=/etc/foxhunt/config.toml

# Source environment configuration
source /etc/foxhunt/environment

3. Service Startup Sequence

Option A: Systemd (Recommended for Production)

# Start core services first
sudo systemctl start foxhunt-database
sudo systemctl start foxhunt-redis

# Start trading services
sudo systemctl start foxhunt-trading-service
sudo systemctl start foxhunt-backtesting-service

# Verify services are running
sudo systemctl status foxhunt-trading-service
sudo systemctl status foxhunt-backtesting-service

Option B: Docker Deployment

# Start via docker-compose
cd /opt/foxhunt
docker-compose up -d

# Verify containers are healthy
docker-compose ps
docker-compose logs -f

Option C: Manual Startup (Development)

# Start trading service
cd /opt/foxhunt
RUST_LOG=info ./target/release/trading-service &

# Start backtesting service
RUST_LOG=info ./target/release/backtesting-service &

# Verify processes
ps aux | grep foxhunt

4. Post-Startup Verification

# Test gRPC connectivity
grpcurl -insecure localhost:50051 list

# Check health endpoints
curl -k https://localhost:50051/health
curl -k https://localhost:50052/health

# Verify TLI client connectivity
cargo run --bin tli-client -- --command ping

---

Configuration Management

Configuration Files

File	Purpose	Location
config.toml	Main service configuration	/etc/foxhunt/config.toml
security.toml	Authentication and TLS settings	/etc/foxhunt/security.toml
logging.toml	Logging configuration	/etc/foxhunt/logging.toml
environment	Environment variables	/etc/foxhunt/environment

Main Configuration Structure

[trading_service]
host = "0.0.0.0"
port = 50051
max_connections = 1000
timeout_seconds = 30

[backtesting_service]
host = "0.0.0.0"
port = 50052
max_connections = 100
timeout_seconds = 300

[database]
url = "postgresql://user:pass@localhost/foxhunt_db"
max_connections = 50
timeout_seconds = 10

[redis]
url = "redis://localhost:6379"
pool_size = 20
timeout_seconds = 5

[security]
tls_cert_path = "/etc/foxhunt/tls/server.crt"
tls_key_path = "/etc/foxhunt/tls/server.key"
ca_cert_path = "/etc/foxhunt/tls/ca.crt"
require_client_cert = true

[rate_limiting]
authenticated_rpm = 1000
api_key_rpm = 5000
trading_burst = 100
window_seconds = 60

[audit]
log_auth_attempts = true
log_trading_operations = true
retention_days = 2555
encrypt_logs = true

Dynamic Configuration Updates

# Update configuration via TLI client
cargo run --bin tli-client -- --command config-update \
  --key "rate_limiting.trading_burst" \
  --value "150"

# Reload configuration without restart
kill -HUP $(pidof trading-service)

# Verify configuration changes
cargo run --bin tli-client -- --command config-get \
  --key "rate_limiting.trading_burst"

Configuration Backup and Restore

# Backup current configuration
cp /etc/foxhunt/config.toml /etc/foxhunt/config.toml.backup.$(date +%Y%m%d)

# Restore configuration
cp /etc/foxhunt/config.toml.backup.20250123 /etc/foxhunt/config.toml
sudo systemctl reload foxhunt-trading-service

---

Daily Operations

Morning Startup Checklist

1. System Health Check

   # Check system status
   cargo run --bin tli-client -- --command system-status
   
   # Verify all services are healthy
   sudo systemctl status foxhunt-*
   

2. Market Data Validation

   # Test market data connectivity
   cargo run --bin tli-client -- --command market-data-test
   
   # Verify real-time feeds
   cargo run --bin tli-client -- --command stream-test --symbols AAPL,GOOGL
   

3. Risk System Verification

   # Check risk limits
   cargo run --bin tli-client -- --command risk-limits-check
   
   # Verify VaR calculations
   cargo run --bin tli-client -- --command var-test
   

End-of-Day Procedures

1. Position Reconciliation

   # Generate position report
   cargo run --bin tli-client -- --command positions-report \
     --format json > /var/log/foxhunt/positions-$(date +%Y%m%d).json
   

2. Performance Summary

   # Generate daily performance report
   cargo run --bin tli-client -- --command performance-summary \
     --date $(date +%Y-%m-%d)
   

3. Log Rotation

   # Rotate application logs
   logrotate /etc/logrotate.d/foxhunt
   
   # Archive audit logs
   /opt/foxhunt/scripts/archive-audit-logs.sh
   

---

Monitoring and Health Checks

Health Check Endpoints

Service	Endpoint	Expected Response
Trading Service	localhost:50051/health	HTTP 200 OK
Backtesting Service	localhost:50052/health	HTTP 200 OK
TLI Client	tli-client --command ping	PONG response

Key Metrics to Monitor

1. Performance Metrics

   • Order submission latency (target: < 50μs)
   • Market data processing latency (target: < 10μs)
   • gRPC request throughput
   • Memory usage and garbage collection

2. Business Metrics

   • Active trading sessions
   • Orders per second
   • Portfolio value-at-risk
   • Risk limit violations

3. System Metrics

   • CPU utilization
   • Memory usage
   • Network throughput
   • Disk I/O

Monitoring Commands

# Real-time metrics dashboard
cargo run --bin tli-client -- --command metrics-dashboard

# Latency monitoring
cargo run --bin tli-client -- --command latency-monitor \
  --interval 5s --duration 1h

# Throughput monitoring
cargo run --bin tli-client -- --command throughput-monitor \
  --service trading --operation submit_order

Alerting Thresholds

Metric	Warning	Critical
Order Latency	> 100μs	> 500μs
CPU Usage	> 70%	> 90%
Memory Usage	> 80%	> 95%
Error Rate	> 1%	> 5%
Risk Violations	Any	Multiple

---

Troubleshooting

Common Issues and Solutions

Issue: "Connection Refused" Error

Symptoms:

Error: Transport error: Connection refused (os error 111)

Diagnosis:

# Check if service is running
ps aux | grep trading-service

# Check port binding
netstat -tlnp | grep 50051

# Check firewall
sudo iptables -L | grep 50051

Solution:

# Restart service
sudo systemctl restart foxhunt-trading-service

# Check logs for startup errors
journalctl -u foxhunt-trading-service -f

Issue: TLS Certificate Errors

Symptoms:

Error: Certificate validation failed: certificate has expired

Diagnosis:

# Check certificate expiration
openssl x509 -in /etc/foxhunt/tls/server.crt -text -noout | grep "Not After"

# Validate certificate chain
openssl verify -CAfile /etc/foxhunt/tls/ca.crt /etc/foxhunt/tls/server.crt

Solution:

# Generate new certificates
/opt/foxhunt/scripts/generate-certificates.sh

# Restart services
sudo systemctl restart foxhunt-trading-service

Issue: High Latency

Symptoms:

• Order submission taking > 100μs
• Market data delays

Diagnosis:

# Check system load
top
iostat 1

# Check network latency
ping trading-venue.com

# Check CPU affinity
taskset -p $(pidof trading-service)

Solution:

# Set CPU affinity for performance
sudo taskset -c 0,1 $(pidof trading-service)

# Increase process priority
sudo renice -10 $(pidof trading-service)

# Check for other processes using CPU
ps aux --sort=-%cpu | head -20

Issue: Authentication Failures

Symptoms:

Error: Access denied: insufficient permissions for trade:execute

Diagnosis:

# Check user permissions
cargo run --bin tli-client -- --command check-permissions \
  --user trader_user_id --permission trade:execute

# Check API key status
cargo run --bin tli-client -- --command api-key-status \
  --key-id abc123

Solution:

# Update user permissions
cargo run --bin tli-client -- --command grant-permission \
  --user trader_user_id --permission trade:execute

# Regenerate API key if expired
cargo run --bin tli-client -- --command create-api-key \
  --user trader_user_id --name "Trading Bot" \
  --permissions trade:execute,order:place

Log Analysis

Application Logs

# View real-time logs
tail -f /var/log/foxhunt/trading-service.log

# Search for errors
grep -i error /var/log/foxhunt/trading-service.log | tail -50

# Filter by timestamp
grep "2025-01-23 14:" /var/log/foxhunt/trading-service.log

Audit Logs

# View authentication attempts
grep "auth_attempt" /var/log/foxhunt/audit.log | tail -20

# View trading operations
grep "trade_operation" /var/log/foxhunt/audit.log | tail -20

# Search for specific user activity
grep "user_id:trader_001" /var/log/foxhunt/audit.log

---

Performance Tuning

Operating System Tuning

# Increase file descriptor limits
echo "* soft nofile 65536" >> /etc/security/limits.conf
echo "* hard nofile 65536" >> /etc/security/limits.conf

# TCP tuning for low latency
echo 'net.core.rmem_max = 134217728' >> /etc/sysctl.conf
echo 'net.core.wmem_max = 134217728' >> /etc/sysctl.conf
echo 'net.ipv4.tcp_rmem = 4096 87380 134217728' >> /etc/sysctl.conf
echo 'net.ipv4.tcp_wmem = 4096 65536 134217728' >> /etc/sysctl.conf
sysctl -p

CPU Affinity and Process Priority

# Set CPU affinity for trading service (cores 0-3)
taskset -c 0-3 $(pidof trading-service)

# Set high priority
renice -15 $(pidof trading-service)

# Isolate CPUs for trading (add to kernel parameters)
# isolcpus=0,1,2,3 nohz_full=0,1,2,3 rcu_nocbs=0,1,2,3

Memory Configuration

# Disable swap for predictable performance
swapoff -a

# Configure huge pages
echo 1024 > /proc/sys/vm/nr_hugepages

# Set memory overcommit
echo 1 > /proc/sys/vm/overcommit_memory

Service-Specific Tuning

Trading Service Configuration

[performance]
connection_pool_size = 100
worker_threads = 16
max_blocking_threads = 512
thread_stack_size = 2097152
enable_thread_pinning = true
enable_numa_binding = true

[latency_optimization]
disable_nagle = true
tcp_nodelay = true
socket_recv_buffer = 1048576
socket_send_buffer = 1048576

Database Tuning

-- PostgreSQL tuning for trading workload
ALTER SYSTEM SET shared_buffers = '8GB';
ALTER SYSTEM SET effective_cache_size = '24GB';
ALTER SYSTEM SET maintenance_work_mem = '2GB';
ALTER SYSTEM SET checkpoint_completion_target = 0.9;
ALTER SYSTEM SET wal_buffers = '16MB';
ALTER SYSTEM SET default_statistics_target = 100;
SELECT pg_reload_conf();

Monitoring Performance

# Monitor latency continuously
cargo run --bin tli-client -- --command latency-monitor \
  --output /var/log/foxhunt/latency.log

# Performance profiling
perf record -g ./target/release/trading-service
perf report

# Memory profiling
valgrind --tool=massif ./target/release/trading-service

---

Emergency Procedures

Emergency Stop Procedures

Immediate Market Stop

# Stop all trading immediately
cargo run --bin tli-client -- --command emergency-stop \
  --type full_shutdown --reason "Market emergency" --confirm

# Cancel all open orders
cargo run --bin tli-client -- --command emergency-stop \
  --type cancel_orders --confirm

# Close all positions
cargo run --bin tli-client -- --command emergency-stop \
  --type close_positions --confirm

Service Emergency Shutdown

# Graceful shutdown with position preservation
sudo systemctl stop foxhunt-trading-service

# Force shutdown if graceful fails
sudo kill -9 $(pidof trading-service)

# Emergency database backup
pg_dump foxhunt_db > /backup/emergency-$(date +%Y%m%d-%H%M%S).sql

Disaster Recovery

Data Backup Verification

# Verify database backup
pg_restore --list /backup/latest-backup.sql

# Test configuration backup
tar -tzf /backup/config-backup.tar.gz

# Verify audit log integrity
/opt/foxhunt/scripts/verify-audit-logs.sh

Service Recovery

# Restore from backup
systemctl stop foxhunt-trading-service
pg_restore -d foxhunt_db /backup/latest-backup.sql
tar -xzf /backup/config-backup.tar.gz -C /etc/foxhunt/
systemctl start foxhunt-trading-service

# Verify recovery
cargo run --bin tli-client -- --command system-status

Incident Response

Security Incident

1. Immediate Actions

   # Revoke all API keys
   cargo run --bin tli-client -- --command revoke-all-api-keys
   
   # Force logout all sessions
   cargo run --bin tli-client -- --command logout-all-sessions
   
   # Block suspicious IPs
   iptables -A INPUT -s <suspicious_ip> -j DROP
   

2. Evidence Collection

   # Backup audit logs
   cp /var/log/foxhunt/audit.log /secure/incident-$(date +%Y%m%d)/
   
   # Capture system state
   ps aux > /secure/incident-$(date +%Y%m%d)/processes.txt
   netstat -tulnp > /secure/incident-$(date +%Y%m%d)/network.txt
   

3. Recovery

   # Restore from clean backup
   # Regenerate all certificates
   # Reset all passwords and API keys
   # Review and update security policies
   

---

Maintenance Procedures

Scheduled Maintenance

Weekly Maintenance

• Certificate rotation check
• Log file rotation and archival
• Performance metrics analysis
• Security audit log review
• Database maintenance (VACUUM, REINDEX)

Monthly Maintenance

• Full system backup verification
• Disaster recovery test
• Security penetration testing
• Performance benchmark comparison
• Dependencies update review

Quarterly Maintenance

• Full security audit
• Certificate renewal
• Hardware performance review
• Disaster recovery full test
• Compliance documentation review

Update Procedures

Binary Updates

# Create backup
cp /opt/foxhunt/target/release/trading-service \
   /backup/trading-service.$(date +%Y%m%d)

# Deploy new binary
sudo systemctl stop foxhunt-trading-service
cp /staging/trading-service /opt/foxhunt/target/release/
sudo systemctl start foxhunt-trading-service

# Verify deployment
cargo run --bin tli-client -- --command version
cargo run --bin tli-client -- --command health-check

Configuration Updates

# Backup current config
cp /etc/foxhunt/config.toml /backup/config.toml.$(date +%Y%m%d)

# Apply new configuration
cp /staging/config.toml /etc/foxhunt/
sudo systemctl reload foxhunt-trading-service

# Verify configuration
cargo run --bin tli-client -- --command config-validate

Database Maintenance

Daily Maintenance

-- Analyze tables for query optimization
ANALYZE;

-- Check for long-running queries
SELECT * FROM pg_stat_activity WHERE state = 'active' AND now() - query_start > interval '1 minute';

Weekly Maintenance

-- Vacuum to reclaim space
VACUUM (ANALYZE, VERBOSE);

-- Reindex for performance
REINDEX DATABASE foxhunt_db;

-- Check database size
SELECT pg_size_pretty(pg_database_size('foxhunt_db'));

Monthly Maintenance

# Full backup
pg_dump foxhunt_db | gzip > /backup/monthly-$(date +%Y%m%d).sql.gz

# Backup verification
pg_restore --list /backup/monthly-$(date +%Y%m%d).sql.gz | head -20

# Archive old audit logs
/opt/foxhunt/scripts/archive-old-logs.sh --older-than 90days

---

Contacts and Escalation

Support Contacts

Role	Contact	Phone	Email
Primary On-Call	Trading Team	+1-555-0100	trading-oncall@company.com
Secondary On-Call	Infrastructure Team	+1-555-0101	infra-oncall@company.com
Database Admin	DBA Team	+1-555-0102	dba@company.com
Security Team	Security Team	+1-555-0103	security@company.com

Escalation Procedures

1. Level 1: Service degradation, non-critical issues
2. Level 2: Service outage, trading impact
3. Level 3: Security incident, data breach
4. Level 4: Regulatory incident, financial loss

Emergency Contacts

• Trading Floor: +1-555-0200
• Risk Management: +1-555-0201
• Compliance: +1-555-0202
• Executive Team: +1-555-0203

---

This operations manual is maintained by the Trading Infrastructure Team. For updates or corrections, please contact: trading-infrastructure@company.com