# Real-Time Streaming System - Enhanced Architecture Design **Date**: 2025-11-02 **Status**: Design Complete **Priority Tiers**: 4 (Immediate → Optional) **Estimated ROI**: 80% value from Priorities 1-2 --- ## Executive Summary This document proposes a **4-tier enhancement roadmap** for the Foxhunt real-time monitoring system, balancing quick wins with long-term improvements. **Design Philosophy**: - **Priority 1 (Immediate)**: Fix broken Rust CLI (2-4 hours) → Unblocks monitoring - **Priority 2 (Short-term)**: Enhanced Python metrics (1-2 days) → 60% value add - **Priority 3 (Medium-term)**: Alert system (3-5 days) → Cost savings - **Priority 4 (Long-term)**: Web dashboard (1-2 weeks) → Nice-to-have **Total Effort**: 2-3 weeks (Priorities 1-3), 4-5 weeks (all priorities) --- ## Architecture Overview ``` ┌─────────────────────────────────────────────────────────────────┐ │ FOXHUNT MONITORING SYSTEM │ └─────────────────────────────────────────────────────────────────┘ ┌─────────────────┐ │ Data Source │ │ (RunPod S3) │ └────────┬────────┘ │ Byte-range │ requests │ (5-10s poll) ▼ ┌─────────────────────────────────────────────────────────────────┐ │ MONITORING LAYER (2 implementations) │ ├─────────────────────────────────┬───────────────────────────────┤ │ Python Monitor (Primary) │ Rust CLI (Secondary/Quick) │ │ - Flexible path handling │ - Single binary │ │ - Rich UI/metrics │ - Fast & portable │ │ - Cost tracking │ - Regex filtering │ │ - Alert system │ - Metrics parsing │ └───────────────┬─────────────────┴──────────────┬────────────────┘ │ │ ▼ ▼ ┌───────────────────────────────┐ ┌──────────────────────────┐ │ Terminal UI Dashboard │ │ Simple Log Stream │ │ - Live metrics table │ │ - Color-coded output │ │ - Cost tracker │ │ - Completion detection │ │ - Trial progress │ └──────────────────────────┘ │ - ETA calculation │ └───────────────┬───────────────┘ │ ▼ ┌───────────────────────────────┐ │ Alert System (P3) │ │ - Error detection │ │ - OOM alerts │ │ - Cost overruns │ │ - Discord/Slack webhooks │ └───────────────┬───────────────┘ │ ▼ ┌───────────────────────────────┐ │ Auto-Termination (P3) │ │ - Save GPU costs │ │ - Safe shutdown │ └───────────────────────────────┘ (Optional) ▼ ┌───────────────────────────────┐ │ Web Dashboard (P4) │ │ - Multi-pod monitoring │ │ - Historical metrics DB │ │ - Real-time charts │ │ - Cost analytics │ └───────────────────────────────┘ ``` --- ## Priority 1: Fix Rust CLI (IMMEDIATE - 2-4 Hours) ### Goal Unblock `foxhunt-deploy monitor` by adding nested path support. ### Problem **File**: `/home/jgrusewski/Work/foxhunt/foxhunt-deploy/src/s3/mod.rs` Current implementation (BROKEN): ```rust pub(crate) async fn list_log_files(&self, pod_id: &str) -> Result> { let prefix = format!("ml_training/{}/", pod_id); // Only searches one level deep } ``` ### Solution A: Add `--run-id` Parameter (RECOMMENDED) **Changes**: 1. **Update CLI args** (`foxhunt-deploy/src/cli/monitor.rs`): ```rust #[derive(Args, Debug)] pub(crate) struct MonitorArgs { /// Pod ID or Run ID to monitor #[arg(required = true)] pub id: String, /// Treat ID as run-id instead of pod-id #[arg(long)] pub run_id: bool, // ... existing args } ``` 2. **Add search function** (`foxhunt-deploy/src/s3/mod.rs`): ```rust pub(crate) async fn find_log_by_run_id(&self, run_id: &str) -> Result> { // Search pattern: ml_training/*/training_runs/{model}/{run_id}/logs/training.log let model_types = ["mamba2", "dqn", "ppo", "tft"]; for model in &model_types { // List all objects with prefix let prefix = format!("ml_training/"); let response = self.client .list_objects_v2() .bucket(&self.bucket) .prefix(&prefix) .delimiter("/") .send() .await?; // Search through outer directories for prefix_obj in response.common_prefixes() { let outer_dir = prefix_obj.prefix(); let log_key = format!( "{}training_runs/{}/{}/logs/training.log", outer_dir, model, run_id ); // Check if this log file exists if self.object_exists(&log_key).await { return Ok(Some(log_key)); } } } Ok(None) } async fn object_exists(&self, key: &str) -> bool { self.client .head_object() .bucket(&self.bucket) .key(key) .send() .await .is_ok() } ``` 3. **Update monitor logic** (`foxhunt-deploy/src/cli/monitor.rs`): ```rust pub(crate) async fn execute(config: &FoxhuntConfig, args: &MonitorArgs) -> Result<()> { let s3_client = S3LogClient::new(&config.s3).await?; let log_file = if args.run_id { // Search by run ID s3_client .find_log_by_run_id(&args.id) .await? .ok_or_else(|| FoxhuntError::S3(format!("No logs found for run_id: {}", args.id)))? } else { // Original pod_id logic (for backwards compatibility) let monitor = LogMonitor::new(s3_client, args.id.clone(), config.s3.poll_interval_secs); monitor.find_log_file().await? .ok_or_else(|| FoxhuntError::S3(format!("No logs found for pod_id: {}", args.id)))? }; // Stream logs from discovered file // ... } ``` **Usage**: ```bash # By run ID (NEW - handles nested paths) ./target/release/foxhunt-deploy monitor run_20251102_210818_hyperopt --run-id --follow # By pod ID (OLD - for backwards compatibility) ./target/release/foxhunt-deploy monitor aryszyyzz3flzo --follow ``` ### Solution B: Recursive Search (FALLBACK) If run-id approach is too complex, make pod-id search recursive: ```rust pub(crate) async fn list_log_files_recursive(&self, pod_id: &str) -> Result> { let mut log_files = Vec::new(); // Try flat structure first (backwards compatibility) let flat_prefix = format!("ml_training/{}/", pod_id); let flat_logs = self.list_objects_with_prefix(&flat_prefix).await?; log_files.extend(flat_logs); // Try nested structure let nested_prefix = "ml_training/"; let all_objects = self.list_objects_recursive(&nested_prefix).await?; // Filter for logs containing pod_id let nested_logs: Vec = all_objects .into_iter() .filter(|key| key.contains(pod_id) && (key.ends_with(".log") || key.ends_with("training.log"))) .collect(); log_files.extend(nested_logs); log_files.sort(); log_files.dedup(); Ok(log_files) } ``` **Pros**: Simple, backwards compatible **Cons**: Slower (scans entire ml_training prefix) ### Testing ```bash # Build cd foxhunt-deploy cargo build --release # Test with completed DQN run cd .. ./target/release/foxhunt-deploy monitor run_20251102_210818_hyperopt --run-id --tail 50 # Expected output: # Found log file: ml_training/.../training_runs/dqn/run_20251102_210818_hyperopt/logs/training.log # [logs displayed] ``` ### Estimated Effort - Solution A (run-id): 2-4 hours - Solution B (recursive): 1-2 hours **Recommendation**: Implement Solution A for better UX and performance. --- ## Priority 2: Enhanced Python Metrics (SHORT-TERM - 1-2 Days) ### Goal Transform `scripts/monitor_logs.py` into a feature-rich monitoring dashboard. ### Component 1: Structured Metrics Extraction **File**: `scripts/monitor_logs.py` (new class) ```python from dataclasses import dataclass from datetime import datetime from typing import Optional import re @dataclass class TrainingMetrics: """Structured training metrics parsed from logs.""" timestamp: datetime epoch: Optional[int] = None total_epochs: Optional[int] = None loss: Optional[float] = None policy_loss: Optional[float] = None # PPO value_loss: Optional[float] = None # PPO q_buy: Optional[float] = None # DQN q_sell: Optional[float] = None # DQN q_hold: Optional[float] = None # DQN episode_reward: Optional[float] = None learning_rate: Optional[float] = None trial_number: Optional[int] = None @classmethod def parse_from_line(cls, line: str, model_type: str) -> Optional['TrainingMetrics']: """Parse metrics from a log line based on model type.""" metrics = cls(timestamp=datetime.now()) if model_type == 'dqn': # Example: "Epoch 10/100 | Loss: 1234.56 | Q(buy): 123.4, Q(sell): 234.5, Q(hold): 345.6 | Reward: 0.123" epoch_match = re.search(r'Epoch\s+(\d+)/(\d+)', line) if epoch_match: metrics.epoch = int(epoch_match.group(1)) metrics.total_epochs = int(epoch_match.group(2)) loss_match = re.search(r'Loss:\s+([\d.]+)', line) if loss_match: metrics.loss = float(loss_match.group(1)) q_buy_match = re.search(r'Q\(buy\):\s+([-\d.]+)', line) if q_buy_match: metrics.q_buy = float(q_buy_match.group(1)) # ... similar for q_sell, q_hold, episode_reward elif model_type == 'ppo': # Example: "Epoch 10 | Policy Loss: 0.123 | Value Loss: 0.456 | Reward: 0.789" epoch_match = re.search(r'Epoch\s+(\d+)', line) if epoch_match: metrics.epoch = int(epoch_match.group(1)) policy_loss_match = re.search(r'Policy Loss:\s+([\d.]+)', line) if policy_loss_match: metrics.policy_loss = float(policy_loss_match.group(1)) value_loss_match = re.search(r'Value Loss:\s+([\d.]+)', line) if value_loss_match: metrics.value_loss = float(value_loss_match.group(1)) # ... similar for episode_reward # Return only if we found at least one metric if any([metrics.epoch, metrics.loss, metrics.q_buy, metrics.policy_loss]): return metrics return None ``` ### Component 2: Cost Tracking ```python from datetime import datetime, timedelta class CostTracker: """Real-time GPU cost tracking.""" def __init__(self, pod_cost_per_hour: float, start_time: datetime): self.pod_cost_per_hour = pod_cost_per_hour self.start_time = start_time def get_elapsed_time(self) -> timedelta: """Get elapsed time since training started.""" return datetime.now() - self.start_time def get_current_cost(self) -> float: """Calculate current cost based on elapsed time.""" elapsed_hours = self.get_elapsed_time().total_seconds() / 3600 return self.pod_cost_per_hour * elapsed_hours def estimate_total_cost(self, trials_completed: int, total_trials: int) -> tuple[float, timedelta]: """ Estimate total cost and time based on current progress. Returns: (estimated_total_cost, estimated_time_remaining) """ if trials_completed == 0: return 0.0, timedelta(0) # Calculate progress rate elapsed = self.get_elapsed_time() progress = trials_completed / total_trials # Estimate total time estimated_total_time = elapsed / progress estimated_remaining = estimated_total_time - elapsed # Estimate total cost total_hours = estimated_total_time.total_seconds() / 3600 estimated_total_cost = self.pod_cost_per_hour * total_hours return estimated_total_cost, estimated_remaining def format_summary(self, trials_completed: int = 0, total_trials: int = 0) -> str: """Format cost summary for display.""" current_cost = self.get_current_cost() elapsed = self.get_elapsed_time() summary = f"💰 Current Cost: ${current_cost:.4f} | ⏱️ Elapsed: {self._format_timedelta(elapsed)}" if trials_completed > 0 and total_trials > 0: est_cost, est_remaining = self.estimate_total_cost(trials_completed, total_trials) summary += f"\n Est. Total: ${est_cost:.4f} | ETA: {self._format_timedelta(est_remaining)}" return summary @staticmethod def _format_timedelta(td: timedelta) -> str: """Format timedelta as human-readable string.""" total_seconds = int(td.total_seconds()) hours, remainder = divmod(total_seconds, 3600) minutes, seconds = divmod(remainder, 60) if hours > 0: return f"{hours}h {minutes}m" elif minutes > 0: return f"{minutes}m {seconds}s" else: return f"{seconds}s" ``` ### Component 3: Terminal UI Dashboard ```python from rich.live import Live from rich.table import Table from rich.panel import Panel from rich.layout import Layout from rich.text import Text class TrainingDashboard: """Live terminal dashboard for training monitoring.""" def __init__(self, run_id: str, model_type: str, cost_tracker: CostTracker): self.run_id = run_id self.model_type = model_type self.cost_tracker = cost_tracker self.metrics_history: list[TrainingMetrics] = [] self.trials_completed = 0 self.total_trials = 0 def add_metrics(self, metrics: TrainingMetrics): """Add new metrics to history.""" self.metrics_history.append(metrics) # Keep only last 20 entries if len(self.metrics_history) > 20: self.metrics_history = self.metrics_history[-20:] def create_layout(self) -> Layout: """Create rich layout with panels.""" layout = Layout() layout.split_column( Layout(name="header", size=5), Layout(name="main", ratio=1), Layout(name="footer", size=3) ) return layout def render_header(self) -> Panel: """Render header panel.""" header_text = Text() header_text.append("🚀 Training Monitor\n", style="bold cyan") header_text.append(f"Run: {self.run_id} | ", style="dim") header_text.append(f"Model: {self.model_type.upper()}", style="bold yellow") return Panel(header_text, border_style="cyan") def render_metrics_table(self) -> Table: """Render metrics table.""" table = Table(title="Recent Training Metrics", box=box.ROUNDED) if self.model_type == 'dqn': table.add_column("Epoch", justify="right", style="cyan") table.add_column("Loss", justify="right", style="yellow") table.add_column("Q(Buy)", justify="right", style="green") table.add_column("Q(Sell)", justify="right", style="red") table.add_column("Q(Hold)", justify="right", style="blue") table.add_column("Reward", justify="right", style="magenta") for m in self.metrics_history[-10:]: # Last 10 entries table.add_row( f"{m.epoch}/{m.total_epochs}" if m.epoch else "-", f"{m.loss:.2f}" if m.loss else "-", f"{m.q_buy:.2f}" if m.q_buy is not None else "-", f"{m.q_sell:.2f}" if m.q_sell is not None else "-", f"{m.q_hold:.2f}" if m.q_hold is not None else "-", f"{m.episode_reward:.4f}" if m.episode_reward else "-" ) elif self.model_type == 'ppo': table.add_column("Epoch", justify="right", style="cyan") table.add_column("Policy Loss", justify="right", style="yellow") table.add_column("Value Loss", justify="right", style="green") table.add_column("Reward", justify="right", style="magenta") for m in self.metrics_history[-10:]: table.add_row( f"{m.epoch}" if m.epoch else "-", f"{m.policy_loss:.4f}" if m.policy_loss else "-", f"{m.value_loss:.4f}" if m.value_loss else "-", f"{m.episode_reward:.4f}" if m.episode_reward else "-" ) return table def render_footer(self) -> Panel: """Render footer with cost tracking.""" footer_text = self.cost_tracker.format_summary( self.trials_completed, self.total_trials ) return Panel(footer_text, border_style="green") def render(self) -> Layout: """Render complete dashboard.""" layout = self.create_layout() layout["header"].update(self.render_header()) layout["main"].update(self.render_metrics_table()) layout["footer"].update(self.render_footer()) return layout ``` ### Component 4: Integration with Existing Monitor **Update `stream_run_logs()` in `scripts/monitor_logs.py`**: ```python def stream_run_logs_with_dashboard( s3_client: S3Client, run_id: str, model_type: str, pod_cost_per_hour: float = 0.25, # RTX A4000 default follow: bool = True, timeout: Optional[int] = None, poll_interval: int = 5 ) -> None: """Stream logs with live dashboard.""" # Initialize components start_time = datetime.now() cost_tracker = CostTracker(pod_cost_per_hour, start_time) dashboard = TrainingDashboard(run_id, model_type, cost_tracker) log_key = f"ml_training/training_runs/{model_type}/{run_id}/logs/training.log" trials_key = f"ml_training/training_runs/{model_type}/{run_id}/hyperopt/trials.json" log_position = 0 with Live(dashboard.render(), refresh_per_second=2) as live: while True: # Tail new log content try: content, log_position = s3_client.tail_log_file(log_key, start_byte=log_position) if content: text = content.decode('utf-8', errors='ignore') lines = text.splitlines() for line in lines: # Parse metrics from line metrics = TrainingMetrics.parse_from_line(line, model_type) if metrics: dashboard.add_metrics(metrics) # Check completion if detect_completion(line): return except S3ObjectNotFoundError: pass # Check trials.json updates try: trials_data = s3_client.download_json(trials_key) dashboard.trials_completed = len(trials_data) except: pass # Refresh dashboard live.update(dashboard.render()) # Check timeout if timeout and (time.time() - start_time.timestamp()) > timeout: break if not follow: break time.sleep(poll_interval) ``` ### Testing ```bash # Activate venv source .venv/bin/activate # Test with completed run python3 scripts/monitor_logs.py --run-id run_20251102_210818_hyperopt --follow # Expected: Live dashboard with metrics table, cost tracking, ETA ``` ### Estimated Effort - Metrics extraction: 4-6 hours - Cost tracking: 2-3 hours - Terminal UI: 4-6 hours - Integration: 2-3 hours - **Total**: 12-18 hours (1.5-2 days) --- ## Priority 3: Alert System (MEDIUM-TERM - 3-5 Days) ### Goal Prevent wasted GPU costs by detecting errors and auto-terminating. ### Component 1: Alert Manager ```python from enum import Enum from typing import Optional, Callable import requests # For Discord/Slack webhooks class AlertSeverity(Enum): INFO = "info" WARNING = "warning" ERROR = "error" CRITICAL = "critical" @dataclass class Alert: severity: AlertSeverity title: str message: str timestamp: datetime run_id: str def format_discord(self) -> dict: """Format as Discord webhook payload.""" color = { AlertSeverity.INFO: 0x00FF00, # Green AlertSeverity.WARNING: 0xFFFF00, # Yellow AlertSeverity.ERROR: 0xFF0000, # Red AlertSeverity.CRITICAL: 0x990000 # Dark red }[self.severity] return { "embeds": [{ "title": f"🚨 {self.title}", "description": self.message, "color": color, "fields": [ {"name": "Run ID", "value": self.run_id, "inline": True}, {"name": "Timestamp", "value": self.timestamp.isoformat(), "inline": True} ] }] } class AlertManager: """Alert system for training monitoring.""" def __init__(self, discord_webhook_url: Optional[str] = None): self.discord_webhook_url = discord_webhook_url self.alerts: list[Alert] = [] # Error patterns self.error_patterns = [ ("CUDA out of memory", AlertSeverity.CRITICAL, "OOM Error"), ("RuntimeError:", AlertSeverity.ERROR, "Runtime Error"), ("AssertionError:", AlertSeverity.ERROR, "Assertion Failed"), ("panic!", AlertSeverity.CRITICAL, "Rust Panic"), ("killed by signal", AlertSeverity.CRITICAL, "Process Killed") ] def check_line(self, line: str, run_id: str) -> Optional[Alert]: """Check log line for alert patterns.""" for pattern, severity, title in self.error_patterns: if pattern in line: alert = Alert( severity=severity, title=title, message=line.strip(), timestamp=datetime.now(), run_id=run_id ) self.alerts.append(alert) return alert return None def check_oom_plateau(self, log_size: int, last_log_size: int, minutes_stalled: int) -> Optional[Alert]: """Detect OOM via log size plateau (no growth).""" if log_size == last_log_size and minutes_stalled > 5: alert = Alert( severity=AlertSeverity.CRITICAL, title="Training Stalled (Possible OOM)", message=f"Log file size unchanged for {minutes_stalled} minutes. Pod may be frozen.", timestamp=datetime.now(), run_id="unknown" ) self.alerts.append(alert) return alert return None def check_cost_overrun(self, current_cost: float, budget: float) -> Optional[Alert]: """Alert when cost exceeds budget.""" if current_cost > budget: alert = Alert( severity=AlertSeverity.WARNING, title="Cost Overrun", message=f"Current cost ${current_cost:.4f} exceeds budget ${budget:.2f}", timestamp=datetime.now(), run_id="unknown" ) self.alerts.append(alert) return alert return None def send_alert(self, alert: Alert): """Send alert to configured channels.""" if self.discord_webhook_url: try: requests.post( self.discord_webhook_url, json=alert.format_discord(), timeout=5 ) except Exception as e: console.print(f"[red]Failed to send Discord alert: {e}[/red]") # Print to console color = { AlertSeverity.INFO: "green", AlertSeverity.WARNING: "yellow", AlertSeverity.ERROR: "red", AlertSeverity.CRITICAL: "bold red" }[alert.severity] console.print(f"[{color}]🚨 {alert.title}: {alert.message}[/{color}]") ``` ### Component 2: Auto-Termination ```python from runpod.client import RunPodClient class AutoTerminator: """Automatic pod termination on completion.""" def __init__(self, client: RunPodClient, pod_id: str, dry_run: bool = False): self.client = client self.pod_id = pod_id self.dry_run = dry_run def should_terminate( self, training_complete: bool, error_detected: bool, cost_exceeded: bool ) -> tuple[bool, str]: """ Determine if pod should be terminated. Returns: (should_terminate, reason) """ if training_complete: return True, "Training completed successfully" if error_detected: return True, "Critical error detected" if cost_exceeded: return True, "Cost budget exceeded" return False, "" def terminate(self, reason: str) -> bool: """Terminate pod with safety checks.""" if self.dry_run: console.print(f"[yellow]DRY RUN: Would terminate pod {self.pod_id} (reason: {reason})[/yellow]") return False # Confirm termination console.print(f"\n[yellow]⚠️ About to terminate pod {self.pod_id}[/yellow]") console.print(f"[yellow]Reason: {reason}[/yellow]") console.print("[dim]Press Enter to confirm, Ctrl+C to cancel...[/dim]") try: input() except KeyboardInterrupt: console.print("\n[green]Termination cancelled[/green]") return False # Terminate pod try: self.client.terminate_pod(self.pod_id) console.print(f"[green]✅ Pod {self.pod_id} terminated[/green]") return True except Exception as e: console.print(f"[red]Failed to terminate pod: {e}[/red]") return False ``` ### Integration ```python def stream_run_logs_with_alerts( s3_client: S3Client, run_id: str, model_type: str, pod_id: str, alert_manager: AlertManager, auto_terminator: AutoTerminator, cost_budget: float = 1.0, # $1 default budget **kwargs ) -> None: """Stream logs with alerts and auto-termination.""" # ... existing monitoring logic ... while True: # Check for alerts for line in new_lines: alert = alert_manager.check_line(line, run_id) if alert: alert_manager.send_alert(alert) # Check OOM plateau oom_alert = alert_manager.check_oom_plateau(current_log_size, last_log_size, minutes_stalled) if oom_alert: alert_manager.send_alert(oom_alert) # Check cost overrun current_cost = cost_tracker.get_current_cost() cost_alert = alert_manager.check_cost_overrun(current_cost, cost_budget) if cost_alert: alert_manager.send_alert(cost_alert) # Check auto-termination should_term, reason = auto_terminator.should_terminate( training_complete, error_detected, current_cost > cost_budget ) if should_term: auto_terminator.terminate(reason) break ``` ### Estimated Effort - Alert manager: 1-2 days - Auto-termination: 1 day - Integration: 1 day - Testing: 1 day - **Total**: 4-5 days --- ## Priority 4: Web Dashboard (LONG-TERM - 1-2 Weeks, OPTIONAL) ### Goal Provide a web-based UI for multi-pod monitoring and historical analytics. ### Architecture ``` ┌─────────────┐ │ Frontend │ (React + Recharts) │ (Port │ - Live metrics charts │ 3000) │ - Multi-pod table └──────┬──────┘ - Cost analytics │ │ WebSocket (socket.io) │ ┌──────▼──────┐ │ Backend │ (FastAPI + Socket.IO) │ (Port │ - S3 polling service │ 8000) │ - Metrics aggregation └──────┬──────┘ - Alert broadcasting │ │ SQLAlchemy ORM │ ┌──────▼──────┐ │ PostgreSQL │ (Historical metrics DB) │ (Port │ - Metrics archive │ 5432) │ - Cost tracking └─────────────┘ - Run metadata ``` ### Database Schema ```sql CREATE TABLE training_runs ( id SERIAL PRIMARY KEY, run_id VARCHAR(255) UNIQUE NOT NULL, model_type VARCHAR(50) NOT NULL, pod_id VARCHAR(255), start_time TIMESTAMP NOT NULL, end_time TIMESTAMP, status VARCHAR(50), -- running, completed, failed total_cost DECIMAL(10, 4), pod_cost_per_hour DECIMAL(10, 4) ); CREATE TABLE training_metrics ( id SERIAL PRIMARY KEY, run_id VARCHAR(255) REFERENCES training_runs(run_id), timestamp TIMESTAMP NOT NULL, epoch INT, loss DECIMAL(15, 6), q_buy DECIMAL(15, 6), q_sell DECIMAL(15, 6), q_hold DECIMAL(15, 6), policy_loss DECIMAL(15, 6), value_loss DECIMAL(15, 6), episode_reward DECIMAL(15, 6), learning_rate DECIMAL(15, 10) ); CREATE TABLE hyperopt_trials ( id SERIAL PRIMARY KEY, run_id VARCHAR(255) REFERENCES training_runs(run_id), trial_number INT NOT NULL, objective_value DECIMAL(15, 6), parameters JSONB, timestamp TIMESTAMP NOT NULL ); CREATE TABLE alerts ( id SERIAL PRIMARY KEY, run_id VARCHAR(255), severity VARCHAR(20), title VARCHAR(255), message TEXT, timestamp TIMESTAMP NOT NULL ); ``` ### Backend Implementation **File**: `monitoring_server/main.py` ```python from fastapi import FastAPI, WebSocket from fastapi.middleware.cors import CORSMiddleware import socketio from sqlalchemy.orm import Session from typing import List import asyncio app = FastAPI() sio = socketio.AsyncServer(async_mode='asgi', cors_allowed_origins='*') socket_app = socketio.ASGIApp(sio, app) # CORS for React frontend app.add_middleware( CORSMiddleware, allow_origins=["http://localhost:3000"], allow_credentials=True, allow_methods=["*"], allow_headers=["*"], ) # Background task: Poll S3 and broadcast metrics async def s3_polling_task(): """Poll S3 for new metrics and broadcast via WebSocket.""" while True: # Get active runs from DB active_runs = get_active_runs() for run in active_runs: # Tail logs from S3 new_metrics = tail_run_logs(run.run_id, run.model_type) if new_metrics: # Save to DB save_metrics(new_metrics) # Broadcast to connected clients await sio.emit('metrics_update', { 'run_id': run.run_id, 'metrics': [m.dict() for m in new_metrics] }) await asyncio.sleep(5) # 5-second poll interval @app.on_event("startup") async def startup_event(): """Start background polling task.""" asyncio.create_task(s3_polling_task()) @app.get("/api/runs") async def get_runs(): """Get all training runs.""" # Query DB return get_all_runs() @app.get("/api/runs/{run_id}/metrics") async def get_run_metrics(run_id: str, limit: int = 100): """Get metrics for a specific run.""" return get_metrics_by_run(run_id, limit) @sio.event async def connect(sid, environ): """Handle WebSocket connection.""" print(f"Client connected: {sid}") @sio.event async def disconnect(sid): """Handle WebSocket disconnection.""" print(f"Client disconnected: {sid}") ``` ### Frontend Implementation **File**: `monitoring_frontend/src/App.tsx` ```typescript import React, { useEffect, useState } from 'react'; import io from 'socket.io-client'; import { LineChart, Line, XAxis, YAxis, CartesianGrid, Tooltip, Legend } from 'recharts'; interface TrainingMetrics { timestamp: string; epoch: number; loss: number; q_buy?: number; q_sell?: number; q_hold?: number; } const socket = io('http://localhost:8000'); function App() { const [metrics, setMetrics] = useState([]); const [runs, setRuns] = useState([]); useEffect(() => { // Fetch initial data fetch('http://localhost:8000/api/runs') .then(res => res.json()) .then(data => setRuns(data)); // Listen for real-time updates socket.on('metrics_update', (data) => { setMetrics(prev => [...prev, ...data.metrics]); }); return () => { socket.off('metrics_update'); }; }, []); return (

Foxhunt Training Monitor

{/* Active runs table */} {runs.map(run => ( ))}
Run ID Model Status Cost Elapsed
{run.run_id} {run.model_type} {run.status} ${run.total_cost} {run.elapsed_time}
{/* Live metrics chart */}
); } export default App; ``` ### Estimated Effort - Database setup + ORM models: 1 day - Backend API + WebSocket: 2-3 days - Frontend components: 2-3 days - Integration + testing: 2 days - **Total**: 7-9 days (1-2 weeks) **Recommendation**: Defer to Phase 2 (Priorities 1-3 provide 80% of value). --- ## Implementation Roadmap ### Phase 1: Quick Wins (1 Week) **Week 1**: - Day 1: Fix Rust CLI (Priority 1) - Days 2-3: Enhanced Python metrics (Priority 2, Part 1) - Days 4-5: Cost tracking + Terminal UI (Priority 2, Part 2) **Deliverables**: - ✅ Working Rust CLI with nested path support - ✅ Python script with live metrics dashboard - ✅ Real-time cost tracking with ETA **Value**: 60% of total value, 20% of total effort --- ### Phase 2: Cost Optimization (1 Week) **Week 2**: - Days 1-3: Alert system (Priority 3, Part 1) - Days 4-5: Auto-termination (Priority 3, Part 2) **Deliverables**: - ✅ Error/OOM alert system - ✅ Discord/Slack integration - ✅ Auto-termination with cost savings **Value**: 20% of total value, 30% of total effort **Expected Cost Savings**: 20-50% reduction in GPU costs (auto-termination prevents "forgotten pods") --- ### Phase 3: Web Dashboard (OPTIONAL - 2 Weeks) **Weeks 3-4**: - Week 3: Backend + database - Week 4: Frontend + integration **Deliverables**: - ✅ Web-based multi-pod monitoring - ✅ Historical metrics database - ✅ Cost analytics and charts **Value**: 20% of total value, 50% of total effort **Recommendation**: Only pursue if Phases 1-2 are highly successful and there's user demand. --- ## Comparison: Polling vs Alternative Approaches ### Option A: S3 Polling (RECOMMENDED - Current Approach) **Pros**: - ✅ Simple infrastructure (no webhooks) - ✅ Works with RunPod S3 - ✅ Can monitor terminated pods - ✅ Byte-range efficiency (99.9% data savings) - ✅ 5-10s latency acceptable **Cons**: - ❌ Slight overhead (repeated requests) - ❌ Not truly real-time (5-10s delay) **Verdict**: Optimal for this use case --- ### Option B: S3 Event Notifications (NOT VIABLE) **Pros**: - ✅ True real-time (sub-second) - ✅ Event-driven (no polling) **Cons**: - ❌ RunPod S3 doesn't support S3 events - ❌ Requires AWS Lambda or webhook endpoint - ❌ More complex error handling **Verdict**: Not possible with RunPod S3 --- ### Option C: RunPod Logs API (NOT RECOMMENDED) **Pros**: - ✅ Official API - ✅ Real-time logs **Cons**: - ❌ Only works while pod is running - ❌ Rate limits - ❌ Higher latency - ❌ Pod restart clears logs **Verdict**: Worse than S3 polling --- ## Cost-Benefit Analysis ### Estimated ROI by Priority | Priority | Effort | Value | ROI | Notes | |----------|--------|-------|-----|-------| | P1: Fix Rust CLI | 2-4 hours | High | **10x** | Unblocks monitoring, minimal effort | | P2: Enhanced Python | 1-2 days | Very High | **5x** | Metrics + cost tracking + UI | | P3: Alert System | 3-5 days | Medium | **3x** | Cost savings from auto-termination | | P4: Web Dashboard | 1-2 weeks | Low | **1x** | Nice-to-have, high effort | **Total Effort (P1-P3)**: 2-3 weeks **Total Value**: 80% of benefits **Recommendation**: Focus on Priorities 1-3. Defer Priority 4 unless there's strong user demand. --- ## Success Metrics ### Priority 1 Success Criteria - ✅ Rust CLI can monitor runs with nested S3 paths - ✅ `--run-id` parameter works correctly - ✅ Backward compatibility maintained (pod-id still works) - ✅ Zero regressions in existing functionality ### Priority 2 Success Criteria - ✅ Metrics extracted correctly (epoch, loss, Q-values, etc.) - ✅ Cost tracking displays live updates - ✅ ETA calculation accurate within 10% - ✅ Terminal UI renders smoothly (no flickering) - ✅ User can monitor training without checking raw logs ### Priority 3 Success Criteria - ✅ Alerts trigger within 10 seconds of error - ✅ Discord/Slack notifications delivered reliably - ✅ Auto-termination saves >20% GPU costs - ✅ Zero false positives (no accidental terminations) - ✅ User can set custom cost budgets ### Priority 4 Success Criteria (Optional) - ✅ Web dashboard supports 5+ concurrent runs - ✅ Real-time updates within 5 seconds - ✅ Historical metrics queryable (30+ days) - ✅ Multi-user support (authentication) --- ## Risks and Mitigations ### Risk 1: Rust CLI Complexity **Risk**: Recursive S3 search may be slow or complex **Mitigation**: Implement Solution A (run-id parameter) with targeted search **Fallback**: Solution B (full recursive search) ### Risk 2: Python Script Dependencies **Risk**: Users forget to activate .venv **Mitigation**: Add clear error messages with setup instructions **Fallback**: Package as standalone binary with PyInstaller ### Risk 3: Alert Fatigue **Risk**: Too many alerts overwhelm users **Mitigation**: Implement severity levels (only send CRITICAL to Discord/Slack) **Fallback**: Add alert suppression logic (max 1 per 5 minutes) ### Risk 4: Auto-Termination Bugs **Risk**: Accidental termination of healthy pods **Mitigation**: Require user confirmation before terminating **Fallback**: Dry-run mode by default, opt-in for auto-termination ### Risk 5: Web Dashboard Scope Creep **Risk**: Priority 4 takes too long, delays other work **Mitigation**: Defer Priority 4 unless Priorities 1-3 succeed **Fallback**: Use simple Terminal UI instead of web dashboard --- ## Conclusion This design provides a clear roadmap for enhancing the Foxhunt monitoring system with **4 priority tiers** balancing quick wins and long-term improvements. **Key Recommendations**: 1. **Implement Priority 1 immediately** (2-4 hours) - Unblocks Rust CLI monitoring 2. **Implement Priority 2 next** (1-2 days) - Provides 60% of total value 3. **Implement Priority 3 if budget allows** (3-5 days) - Saves 20-50% GPU costs 4. **Defer Priority 4** (1-2 weeks) - Only if strong user demand **Expected Outcomes**: - ✅ Real-time monitoring with live metrics - ✅ Cost tracking and auto-termination - ✅ 20-50% reduction in GPU costs - ✅ Better UX for training runs **Total ROI**: 5-10x improvement in monitoring capabilities with 2-3 weeks of effort (Priorities 1-3). **Next Steps**: Proceed to implementation with Priority 1 quick fix.