foxhunt/REALTIME_STREAMING_DESIGN.md

Real-Time Streaming System - Enhanced Architecture Design

Date: 2025-11-02 Status: Design Complete Priority Tiers: 4 (Immediate → Optional) Estimated ROI: 80% value from Priorities 1-2

---

Executive Summary

This document proposes a 4-tier enhancement roadmap for the Foxhunt real-time monitoring system, balancing quick wins with long-term improvements.

Design Philosophy:

• Priority 1 (Immediate): Fix broken Rust CLI (2-4 hours) → Unblocks monitoring
• Priority 2 (Short-term): Enhanced Python metrics (1-2 days) → 60% value add
• Priority 3 (Medium-term): Alert system (3-5 days) → Cost savings
• Priority 4 (Long-term): Web dashboard (1-2 weeks) → Nice-to-have

Total Effort: 2-3 weeks (Priorities 1-3), 4-5 weeks (all priorities)

---

Architecture Overview

┌─────────────────────────────────────────────────────────────────┐
│                    FOXHUNT MONITORING SYSTEM                      │
└─────────────────────────────────────────────────────────────────┘

┌─────────────────┐
│   Data Source   │
│   (RunPod S3)   │
└────────┬────────┘
         │ Byte-range
         │ requests
         │ (5-10s poll)
         ▼
┌─────────────────────────────────────────────────────────────────┐
│              MONITORING LAYER (2 implementations)                 │
├─────────────────────────────────┬───────────────────────────────┤
│     Python Monitor (Primary)    │   Rust CLI (Secondary/Quick)  │
│  - Flexible path handling       │  - Single binary              │
│  - Rich UI/metrics              │  - Fast & portable            │
│  - Cost tracking                │  - Regex filtering            │
│  - Alert system                 │  - Metrics parsing            │
└───────────────┬─────────────────┴──────────────┬────────────────┘
                │                                │
                ▼                                ▼
┌───────────────────────────────┐    ┌──────────────────────────┐
│  Terminal UI Dashboard        │    │  Simple Log Stream       │
│  - Live metrics table         │    │  - Color-coded output    │
│  - Cost tracker               │    │  - Completion detection  │
│  - Trial progress             │    └──────────────────────────┘
│  - ETA calculation            │
└───────────────┬───────────────┘
                │
                ▼
┌───────────────────────────────┐
│      Alert System (P3)        │
│  - Error detection            │
│  - OOM alerts                 │
│  - Cost overruns              │
│  - Discord/Slack webhooks     │
└───────────────┬───────────────┘
                │
                ▼
┌───────────────────────────────┐
│   Auto-Termination (P3)       │
│  - Save GPU costs             │
│  - Safe shutdown              │
└───────────────────────────────┘

                (Optional)
                    ▼
┌───────────────────────────────┐
│    Web Dashboard (P4)         │
│  - Multi-pod monitoring       │
│  - Historical metrics DB      │
│  - Real-time charts           │
│  - Cost analytics             │
└───────────────────────────────┘

---

Priority 1: Fix Rust CLI (IMMEDIATE - 2-4 Hours)

Goal

Unblock foxhunt-deploy monitor by adding nested path support.

Problem

File: /home/jgrusewski/Work/foxhunt/foxhunt-deploy/src/s3/mod.rs

Current implementation (BROKEN):

pub(crate) async fn list_log_files(&self, pod_id: &str) -> Result<Vec<String>> {
    let prefix = format!("ml_training/{}/", pod_id);
    // Only searches one level deep
}

Solution A: Add --run-id Parameter (RECOMMENDED)

Changes:

1. Update CLI args (foxhunt-deploy/src/cli/monitor.rs):

#[derive(Args, Debug)]
pub(crate) struct MonitorArgs {
    /// Pod ID or Run ID to monitor
    #[arg(required = true)]
    pub id: String,

    /// Treat ID as run-id instead of pod-id
    #[arg(long)]
    pub run_id: bool,

    // ... existing args
}

2. Add search function (foxhunt-deploy/src/s3/mod.rs):

pub(crate) async fn find_log_by_run_id(&self, run_id: &str) -> Result<Option<String>> {
    // Search pattern: ml_training/*/training_runs/{model}/{run_id}/logs/training.log
    let model_types = ["mamba2", "dqn", "ppo", "tft"];

    for model in &model_types {
        // List all objects with prefix
        let prefix = format!("ml_training/");
        let response = self.client
            .list_objects_v2()
            .bucket(&self.bucket)
            .prefix(&prefix)
            .delimiter("/")
            .send()
            .await?;

        // Search through outer directories
        for prefix_obj in response.common_prefixes() {
            let outer_dir = prefix_obj.prefix();
            let log_key = format!(
                "{}training_runs/{}/{}/logs/training.log",
                outer_dir, model, run_id
            );

            // Check if this log file exists
            if self.object_exists(&log_key).await {
                return Ok(Some(log_key));
            }
        }
    }

    Ok(None)
}

async fn object_exists(&self, key: &str) -> bool {
    self.client
        .head_object()
        .bucket(&self.bucket)
        .key(key)
        .send()
        .await
        .is_ok()
}

3. Update monitor logic (foxhunt-deploy/src/cli/monitor.rs):

pub(crate) async fn execute(config: &FoxhuntConfig, args: &MonitorArgs) -> Result<()> {
    let s3_client = S3LogClient::new(&config.s3).await?;

    let log_file = if args.run_id {
        // Search by run ID
        s3_client
            .find_log_by_run_id(&args.id)
            .await?
            .ok_or_else(|| FoxhuntError::S3(format!("No logs found for run_id: {}", args.id)))?
    } else {
        // Original pod_id logic (for backwards compatibility)
        let monitor = LogMonitor::new(s3_client, args.id.clone(), config.s3.poll_interval_secs);
        monitor.find_log_file().await?
            .ok_or_else(|| FoxhuntError::S3(format!("No logs found for pod_id: {}", args.id)))?
    };

    // Stream logs from discovered file
    // ...
}

Usage:

# By run ID (NEW - handles nested paths)
./target/release/foxhunt-deploy monitor run_20251102_210818_hyperopt --run-id --follow

# By pod ID (OLD - for backwards compatibility)
./target/release/foxhunt-deploy monitor aryszyyzz3flzo --follow

Solution B: Recursive Search (FALLBACK)

If run-id approach is too complex, make pod-id search recursive:

pub(crate) async fn list_log_files_recursive(&self, pod_id: &str) -> Result<Vec<String>> {
    let mut log_files = Vec::new();

    // Try flat structure first (backwards compatibility)
    let flat_prefix = format!("ml_training/{}/", pod_id);
    let flat_logs = self.list_objects_with_prefix(&flat_prefix).await?;
    log_files.extend(flat_logs);

    // Try nested structure
    let nested_prefix = "ml_training/";
    let all_objects = self.list_objects_recursive(&nested_prefix).await?;

    // Filter for logs containing pod_id
    let nested_logs: Vec<String> = all_objects
        .into_iter()
        .filter(|key| key.contains(pod_id) && (key.ends_with(".log") || key.ends_with("training.log")))
        .collect();

    log_files.extend(nested_logs);
    log_files.sort();
    log_files.dedup();

    Ok(log_files)
}

Pros: Simple, backwards compatible Cons: Slower (scans entire ml_training prefix)

Testing

# Build
cd foxhunt-deploy
cargo build --release

# Test with completed DQN run
cd ..
./target/release/foxhunt-deploy monitor run_20251102_210818_hyperopt --run-id --tail 50

# Expected output:
# Found log file: ml_training/.../training_runs/dqn/run_20251102_210818_hyperopt/logs/training.log
# [logs displayed]

Estimated Effort

• Solution A (run-id): 2-4 hours
• Solution B (recursive): 1-2 hours

Recommendation: Implement Solution A for better UX and performance.

---

Priority 2: Enhanced Python Metrics (SHORT-TERM - 1-2 Days)

Goal

Transform scripts/monitor_logs.py into a feature-rich monitoring dashboard.

Component 1: Structured Metrics Extraction

File: scripts/monitor_logs.py (new class)

from dataclasses import dataclass
from datetime import datetime
from typing import Optional
import re

@dataclass
class TrainingMetrics:
    """Structured training metrics parsed from logs."""
    timestamp: datetime
    epoch: Optional[int] = None
    total_epochs: Optional[int] = None
    loss: Optional[float] = None
    policy_loss: Optional[float] = None  # PPO
    value_loss: Optional[float] = None   # PPO
    q_buy: Optional[float] = None        # DQN
    q_sell: Optional[float] = None       # DQN
    q_hold: Optional[float] = None       # DQN
    episode_reward: Optional[float] = None
    learning_rate: Optional[float] = None
    trial_number: Optional[int] = None

    @classmethod
    def parse_from_line(cls, line: str, model_type: str) -> Optional['TrainingMetrics']:
        """Parse metrics from a log line based on model type."""
        metrics = cls(timestamp=datetime.now())

        if model_type == 'dqn':
            # Example: "Epoch 10/100 | Loss: 1234.56 | Q(buy): 123.4, Q(sell): 234.5, Q(hold): 345.6 | Reward: 0.123"
            epoch_match = re.search(r'Epoch\s+(\d+)/(\d+)', line)
            if epoch_match:
                metrics.epoch = int(epoch_match.group(1))
                metrics.total_epochs = int(epoch_match.group(2))

            loss_match = re.search(r'Loss:\s+([\d.]+)', line)
            if loss_match:
                metrics.loss = float(loss_match.group(1))

            q_buy_match = re.search(r'Q\(buy\):\s+([-\d.]+)', line)
            if q_buy_match:
                metrics.q_buy = float(q_buy_match.group(1))

            # ... similar for q_sell, q_hold, episode_reward

        elif model_type == 'ppo':
            # Example: "Epoch 10 | Policy Loss: 0.123 | Value Loss: 0.456 | Reward: 0.789"
            epoch_match = re.search(r'Epoch\s+(\d+)', line)
            if epoch_match:
                metrics.epoch = int(epoch_match.group(1))

            policy_loss_match = re.search(r'Policy Loss:\s+([\d.]+)', line)
            if policy_loss_match:
                metrics.policy_loss = float(policy_loss_match.group(1))

            value_loss_match = re.search(r'Value Loss:\s+([\d.]+)', line)
            if value_loss_match:
                metrics.value_loss = float(value_loss_match.group(1))

            # ... similar for episode_reward

        # Return only if we found at least one metric
        if any([metrics.epoch, metrics.loss, metrics.q_buy, metrics.policy_loss]):
            return metrics
        return None

Component 2: Cost Tracking

from datetime import datetime, timedelta

class CostTracker:
    """Real-time GPU cost tracking."""

    def __init__(self, pod_cost_per_hour: float, start_time: datetime):
        self.pod_cost_per_hour = pod_cost_per_hour
        self.start_time = start_time

    def get_elapsed_time(self) -> timedelta:
        """Get elapsed time since training started."""
        return datetime.now() - self.start_time

    def get_current_cost(self) -> float:
        """Calculate current cost based on elapsed time."""
        elapsed_hours = self.get_elapsed_time().total_seconds() / 3600
        return self.pod_cost_per_hour * elapsed_hours

    def estimate_total_cost(self, trials_completed: int, total_trials: int) -> tuple[float, timedelta]:
        """
        Estimate total cost and time based on current progress.

        Returns:
            (estimated_total_cost, estimated_time_remaining)
        """
        if trials_completed == 0:
            return 0.0, timedelta(0)

        # Calculate progress rate
        elapsed = self.get_elapsed_time()
        progress = trials_completed / total_trials

        # Estimate total time
        estimated_total_time = elapsed / progress
        estimated_remaining = estimated_total_time - elapsed

        # Estimate total cost
        total_hours = estimated_total_time.total_seconds() / 3600
        estimated_total_cost = self.pod_cost_per_hour * total_hours

        return estimated_total_cost, estimated_remaining

    def format_summary(self, trials_completed: int = 0, total_trials: int = 0) -> str:
        """Format cost summary for display."""
        current_cost = self.get_current_cost()
        elapsed = self.get_elapsed_time()

        summary = f"💰 Current Cost: ${current_cost:.4f} | ⏱️  Elapsed: {self._format_timedelta(elapsed)}"

        if trials_completed > 0 and total_trials > 0:
            est_cost, est_remaining = self.estimate_total_cost(trials_completed, total_trials)
            summary += f"\n   Est. Total: ${est_cost:.4f} | ETA: {self._format_timedelta(est_remaining)}"

        return summary

    @staticmethod
    def _format_timedelta(td: timedelta) -> str:
        """Format timedelta as human-readable string."""
        total_seconds = int(td.total_seconds())
        hours, remainder = divmod(total_seconds, 3600)
        minutes, seconds = divmod(remainder, 60)

        if hours > 0:
            return f"{hours}h {minutes}m"
        elif minutes > 0:
            return f"{minutes}m {seconds}s"
        else:
            return f"{seconds}s"

Component 3: Terminal UI Dashboard

from rich.live import Live
from rich.table import Table
from rich.panel import Panel
from rich.layout import Layout
from rich.text import Text

class TrainingDashboard:
    """Live terminal dashboard for training monitoring."""

    def __init__(self, run_id: str, model_type: str, cost_tracker: CostTracker):
        self.run_id = run_id
        self.model_type = model_type
        self.cost_tracker = cost_tracker
        self.metrics_history: list[TrainingMetrics] = []
        self.trials_completed = 0
        self.total_trials = 0

    def add_metrics(self, metrics: TrainingMetrics):
        """Add new metrics to history."""
        self.metrics_history.append(metrics)
        # Keep only last 20 entries
        if len(self.metrics_history) > 20:
            self.metrics_history = self.metrics_history[-20:]

    def create_layout(self) -> Layout:
        """Create rich layout with panels."""
        layout = Layout()

        layout.split_column(
            Layout(name="header", size=5),
            Layout(name="main", ratio=1),
            Layout(name="footer", size=3)
        )

        return layout

    def render_header(self) -> Panel:
        """Render header panel."""
        header_text = Text()
        header_text.append("🚀 Training Monitor\n", style="bold cyan")
        header_text.append(f"Run: {self.run_id} | ", style="dim")
        header_text.append(f"Model: {self.model_type.upper()}", style="bold yellow")

        return Panel(header_text, border_style="cyan")

    def render_metrics_table(self) -> Table:
        """Render metrics table."""
        table = Table(title="Recent Training Metrics", box=box.ROUNDED)

        if self.model_type == 'dqn':
            table.add_column("Epoch", justify="right", style="cyan")
            table.add_column("Loss", justify="right", style="yellow")
            table.add_column("Q(Buy)", justify="right", style="green")
            table.add_column("Q(Sell)", justify="right", style="red")
            table.add_column("Q(Hold)", justify="right", style="blue")
            table.add_column("Reward", justify="right", style="magenta")

            for m in self.metrics_history[-10:]:  # Last 10 entries
                table.add_row(
                    f"{m.epoch}/{m.total_epochs}" if m.epoch else "-",
                    f"{m.loss:.2f}" if m.loss else "-",
                    f"{m.q_buy:.2f}" if m.q_buy is not None else "-",
                    f"{m.q_sell:.2f}" if m.q_sell is not None else "-",
                    f"{m.q_hold:.2f}" if m.q_hold is not None else "-",
                    f"{m.episode_reward:.4f}" if m.episode_reward else "-"
                )

        elif self.model_type == 'ppo':
            table.add_column("Epoch", justify="right", style="cyan")
            table.add_column("Policy Loss", justify="right", style="yellow")
            table.add_column("Value Loss", justify="right", style="green")
            table.add_column("Reward", justify="right", style="magenta")

            for m in self.metrics_history[-10:]:
                table.add_row(
                    f"{m.epoch}" if m.epoch else "-",
                    f"{m.policy_loss:.4f}" if m.policy_loss else "-",
                    f"{m.value_loss:.4f}" if m.value_loss else "-",
                    f"{m.episode_reward:.4f}" if m.episode_reward else "-"
                )

        return table

    def render_footer(self) -> Panel:
        """Render footer with cost tracking."""
        footer_text = self.cost_tracker.format_summary(
            self.trials_completed,
            self.total_trials
        )
        return Panel(footer_text, border_style="green")

    def render(self) -> Layout:
        """Render complete dashboard."""
        layout = self.create_layout()
        layout["header"].update(self.render_header())
        layout["main"].update(self.render_metrics_table())
        layout["footer"].update(self.render_footer())
        return layout

Component 4: Integration with Existing Monitor

Update stream_run_logs() in scripts/monitor_logs.py:

def stream_run_logs_with_dashboard(
    s3_client: S3Client,
    run_id: str,
    model_type: str,
    pod_cost_per_hour: float = 0.25,  # RTX A4000 default
    follow: bool = True,
    timeout: Optional[int] = None,
    poll_interval: int = 5
) -> None:
    """Stream logs with live dashboard."""

    # Initialize components
    start_time = datetime.now()
    cost_tracker = CostTracker(pod_cost_per_hour, start_time)
    dashboard = TrainingDashboard(run_id, model_type, cost_tracker)

    log_key = f"ml_training/training_runs/{model_type}/{run_id}/logs/training.log"
    trials_key = f"ml_training/training_runs/{model_type}/{run_id}/hyperopt/trials.json"

    log_position = 0

    with Live(dashboard.render(), refresh_per_second=2) as live:
        while True:
            # Tail new log content
            try:
                content, log_position = s3_client.tail_log_file(log_key, start_byte=log_position)

                if content:
                    text = content.decode('utf-8', errors='ignore')
                    lines = text.splitlines()

                    for line in lines:
                        # Parse metrics from line
                        metrics = TrainingMetrics.parse_from_line(line, model_type)
                        if metrics:
                            dashboard.add_metrics(metrics)

                        # Check completion
                        if detect_completion(line):
                            return

            except S3ObjectNotFoundError:
                pass

            # Check trials.json updates
            try:
                trials_data = s3_client.download_json(trials_key)
                dashboard.trials_completed = len(trials_data)
            except:
                pass

            # Refresh dashboard
            live.update(dashboard.render())

            # Check timeout
            if timeout and (time.time() - start_time.timestamp()) > timeout:
                break

            if not follow:
                break

            time.sleep(poll_interval)

Testing

# Activate venv
source .venv/bin/activate

# Test with completed run
python3 scripts/monitor_logs.py --run-id run_20251102_210818_hyperopt --follow

# Expected: Live dashboard with metrics table, cost tracking, ETA

Estimated Effort

• Metrics extraction: 4-6 hours
• Cost tracking: 2-3 hours
• Terminal UI: 4-6 hours
• Integration: 2-3 hours
• Total: 12-18 hours (1.5-2 days)

---

Priority 3: Alert System (MEDIUM-TERM - 3-5 Days)

Goal

Prevent wasted GPU costs by detecting errors and auto-terminating.

Component 1: Alert Manager

from enum import Enum
from typing import Optional, Callable
import requests  # For Discord/Slack webhooks

class AlertSeverity(Enum):
    INFO = "info"
    WARNING = "warning"
    ERROR = "error"
    CRITICAL = "critical"

@dataclass
class Alert:
    severity: AlertSeverity
    title: str
    message: str
    timestamp: datetime
    run_id: str

    def format_discord(self) -> dict:
        """Format as Discord webhook payload."""
        color = {
            AlertSeverity.INFO: 0x00FF00,      # Green
            AlertSeverity.WARNING: 0xFFFF00,   # Yellow
            AlertSeverity.ERROR: 0xFF0000,     # Red
            AlertSeverity.CRITICAL: 0x990000   # Dark red
        }[self.severity]

        return {
            "embeds": [{
                "title": f"🚨 {self.title}",
                "description": self.message,
                "color": color,
                "fields": [
                    {"name": "Run ID", "value": self.run_id, "inline": True},
                    {"name": "Timestamp", "value": self.timestamp.isoformat(), "inline": True}
                ]
            }]
        }

class AlertManager:
    """Alert system for training monitoring."""

    def __init__(self, discord_webhook_url: Optional[str] = None):
        self.discord_webhook_url = discord_webhook_url
        self.alerts: list[Alert] = []

        # Error patterns
        self.error_patterns = [
            ("CUDA out of memory", AlertSeverity.CRITICAL, "OOM Error"),
            ("RuntimeError:", AlertSeverity.ERROR, "Runtime Error"),
            ("AssertionError:", AlertSeverity.ERROR, "Assertion Failed"),
            ("panic!", AlertSeverity.CRITICAL, "Rust Panic"),
            ("killed by signal", AlertSeverity.CRITICAL, "Process Killed")
        ]

    def check_line(self, line: str, run_id: str) -> Optional[Alert]:
        """Check log line for alert patterns."""
        for pattern, severity, title in self.error_patterns:
            if pattern in line:
                alert = Alert(
                    severity=severity,
                    title=title,
                    message=line.strip(),
                    timestamp=datetime.now(),
                    run_id=run_id
                )
                self.alerts.append(alert)
                return alert
        return None

    def check_oom_plateau(self, log_size: int, last_log_size: int, minutes_stalled: int) -> Optional[Alert]:
        """Detect OOM via log size plateau (no growth)."""
        if log_size == last_log_size and minutes_stalled > 5:
            alert = Alert(
                severity=AlertSeverity.CRITICAL,
                title="Training Stalled (Possible OOM)",
                message=f"Log file size unchanged for {minutes_stalled} minutes. Pod may be frozen.",
                timestamp=datetime.now(),
                run_id="unknown"
            )
            self.alerts.append(alert)
            return alert
        return None

    def check_cost_overrun(self, current_cost: float, budget: float) -> Optional[Alert]:
        """Alert when cost exceeds budget."""
        if current_cost > budget:
            alert = Alert(
                severity=AlertSeverity.WARNING,
                title="Cost Overrun",
                message=f"Current cost ${current_cost:.4f} exceeds budget ${budget:.2f}",
                timestamp=datetime.now(),
                run_id="unknown"
            )
            self.alerts.append(alert)
            return alert
        return None

    def send_alert(self, alert: Alert):
        """Send alert to configured channels."""
        if self.discord_webhook_url:
            try:
                requests.post(
                    self.discord_webhook_url,
                    json=alert.format_discord(),
                    timeout=5
                )
            except Exception as e:
                console.print(f"[red]Failed to send Discord alert: {e}[/red]")

        # Print to console
        color = {
            AlertSeverity.INFO: "green",
            AlertSeverity.WARNING: "yellow",
            AlertSeverity.ERROR: "red",
            AlertSeverity.CRITICAL: "bold red"
        }[alert.severity]

        console.print(f"[{color}]🚨 {alert.title}: {alert.message}[/{color}]")

Component 2: Auto-Termination

from runpod.client import RunPodClient

class AutoTerminator:
    """Automatic pod termination on completion."""

    def __init__(self, client: RunPodClient, pod_id: str, dry_run: bool = False):
        self.client = client
        self.pod_id = pod_id
        self.dry_run = dry_run

    def should_terminate(
        self,
        training_complete: bool,
        error_detected: bool,
        cost_exceeded: bool
    ) -> tuple[bool, str]:
        """
        Determine if pod should be terminated.

        Returns:
            (should_terminate, reason)
        """
        if training_complete:
            return True, "Training completed successfully"

        if error_detected:
            return True, "Critical error detected"

        if cost_exceeded:
            return True, "Cost budget exceeded"

        return False, ""

    def terminate(self, reason: str) -> bool:
        """Terminate pod with safety checks."""
        if self.dry_run:
            console.print(f"[yellow]DRY RUN: Would terminate pod {self.pod_id} (reason: {reason})[/yellow]")
            return False

        # Confirm termination
        console.print(f"\n[yellow]⚠️  About to terminate pod {self.pod_id}[/yellow]")
        console.print(f"[yellow]Reason: {reason}[/yellow]")
        console.print("[dim]Press Enter to confirm, Ctrl+C to cancel...[/dim]")

        try:
            input()
        except KeyboardInterrupt:
            console.print("\n[green]Termination cancelled[/green]")
            return False

        # Terminate pod
        try:
            self.client.terminate_pod(self.pod_id)
            console.print(f"[green]✅ Pod {self.pod_id} terminated[/green]")
            return True
        except Exception as e:
            console.print(f"[red]Failed to terminate pod: {e}[/red]")
            return False

Integration

def stream_run_logs_with_alerts(
    s3_client: S3Client,
    run_id: str,
    model_type: str,
    pod_id: str,
    alert_manager: AlertManager,
    auto_terminator: AutoTerminator,
    cost_budget: float = 1.0,  # $1 default budget
    **kwargs
) -> None:
    """Stream logs with alerts and auto-termination."""

    # ... existing monitoring logic ...

    while True:
        # Check for alerts
        for line in new_lines:
            alert = alert_manager.check_line(line, run_id)
            if alert:
                alert_manager.send_alert(alert)

        # Check OOM plateau
        oom_alert = alert_manager.check_oom_plateau(current_log_size, last_log_size, minutes_stalled)
        if oom_alert:
            alert_manager.send_alert(oom_alert)

        # Check cost overrun
        current_cost = cost_tracker.get_current_cost()
        cost_alert = alert_manager.check_cost_overrun(current_cost, cost_budget)
        if cost_alert:
            alert_manager.send_alert(cost_alert)

        # Check auto-termination
        should_term, reason = auto_terminator.should_terminate(
            training_complete,
            error_detected,
            current_cost > cost_budget
        )

        if should_term:
            auto_terminator.terminate(reason)
            break

Estimated Effort

• Alert manager: 1-2 days
• Auto-termination: 1 day
• Integration: 1 day
• Testing: 1 day
• Total: 4-5 days

---

Priority 4: Web Dashboard (LONG-TERM - 1-2 Weeks, OPTIONAL)

Goal

Provide a web-based UI for multi-pod monitoring and historical analytics.

Architecture

┌─────────────┐
│   Frontend  │  (React + Recharts)
│   (Port     │  - Live metrics charts
│    3000)    │  - Multi-pod table
└──────┬──────┘  - Cost analytics
       │
       │ WebSocket (socket.io)
       │
┌──────▼──────┐
│   Backend   │  (FastAPI + Socket.IO)
│   (Port     │  - S3 polling service
│    8000)    │  - Metrics aggregation
└──────┬──────┘  - Alert broadcasting
       │
       │ SQLAlchemy ORM
       │
┌──────▼──────┐
│  PostgreSQL │  (Historical metrics DB)
│   (Port     │  - Metrics archive
│    5432)    │  - Cost tracking
└─────────────┘  - Run metadata

Database Schema

CREATE TABLE training_runs (
    id SERIAL PRIMARY KEY,
    run_id VARCHAR(255) UNIQUE NOT NULL,
    model_type VARCHAR(50) NOT NULL,
    pod_id VARCHAR(255),
    start_time TIMESTAMP NOT NULL,
    end_time TIMESTAMP,
    status VARCHAR(50),  -- running, completed, failed
    total_cost DECIMAL(10, 4),
    pod_cost_per_hour DECIMAL(10, 4)
);

CREATE TABLE training_metrics (
    id SERIAL PRIMARY KEY,
    run_id VARCHAR(255) REFERENCES training_runs(run_id),
    timestamp TIMESTAMP NOT NULL,
    epoch INT,
    loss DECIMAL(15, 6),
    q_buy DECIMAL(15, 6),
    q_sell DECIMAL(15, 6),
    q_hold DECIMAL(15, 6),
    policy_loss DECIMAL(15, 6),
    value_loss DECIMAL(15, 6),
    episode_reward DECIMAL(15, 6),
    learning_rate DECIMAL(15, 10)
);

CREATE TABLE hyperopt_trials (
    id SERIAL PRIMARY KEY,
    run_id VARCHAR(255) REFERENCES training_runs(run_id),
    trial_number INT NOT NULL,
    objective_value DECIMAL(15, 6),
    parameters JSONB,
    timestamp TIMESTAMP NOT NULL
);

CREATE TABLE alerts (
    id SERIAL PRIMARY KEY,
    run_id VARCHAR(255),
    severity VARCHAR(20),
    title VARCHAR(255),
    message TEXT,
    timestamp TIMESTAMP NOT NULL
);

Backend Implementation

File: monitoring_server/main.py

from fastapi import FastAPI, WebSocket
from fastapi.middleware.cors import CORSMiddleware
import socketio
from sqlalchemy.orm import Session
from typing import List
import asyncio

app = FastAPI()
sio = socketio.AsyncServer(async_mode='asgi', cors_allowed_origins='*')
socket_app = socketio.ASGIApp(sio, app)

# CORS for React frontend
app.add_middleware(
    CORSMiddleware,
    allow_origins=["http://localhost:3000"],
    allow_credentials=True,
    allow_methods=["*"],
    allow_headers=["*"],
)

# Background task: Poll S3 and broadcast metrics
async def s3_polling_task():
    """Poll S3 for new metrics and broadcast via WebSocket."""
    while True:
        # Get active runs from DB
        active_runs = get_active_runs()

        for run in active_runs:
            # Tail logs from S3
            new_metrics = tail_run_logs(run.run_id, run.model_type)

            if new_metrics:
                # Save to DB
                save_metrics(new_metrics)

                # Broadcast to connected clients
                await sio.emit('metrics_update', {
                    'run_id': run.run_id,
                    'metrics': [m.dict() for m in new_metrics]
                })

        await asyncio.sleep(5)  # 5-second poll interval

@app.on_event("startup")
async def startup_event():
    """Start background polling task."""
    asyncio.create_task(s3_polling_task())

@app.get("/api/runs")
async def get_runs():
    """Get all training runs."""
    # Query DB
    return get_all_runs()

@app.get("/api/runs/{run_id}/metrics")
async def get_run_metrics(run_id: str, limit: int = 100):
    """Get metrics for a specific run."""
    return get_metrics_by_run(run_id, limit)

@sio.event
async def connect(sid, environ):
    """Handle WebSocket connection."""
    print(f"Client connected: {sid}")

@sio.event
async def disconnect(sid):
    """Handle WebSocket disconnection."""
    print(f"Client disconnected: {sid}")

Frontend Implementation

File: monitoring_frontend/src/App.tsx

import React, { useEffect, useState } from 'react';
import io from 'socket.io-client';
import { LineChart, Line, XAxis, YAxis, CartesianGrid, Tooltip, Legend } from 'recharts';

interface TrainingMetrics {
  timestamp: string;
  epoch: number;
  loss: number;
  q_buy?: number;
  q_sell?: number;
  q_hold?: number;
}

const socket = io('http://localhost:8000');

function App() {
  const [metrics, setMetrics] = useState<TrainingMetrics[]>([]);
  const [runs, setRuns] = useState([]);

  useEffect(() => {
    // Fetch initial data
    fetch('http://localhost:8000/api/runs')
      .then(res => res.json())
      .then(data => setRuns(data));

    // Listen for real-time updates
    socket.on('metrics_update', (data) => {
      setMetrics(prev => [...prev, ...data.metrics]);
    });

    return () => {
      socket.off('metrics_update');
    };
  }, []);

  return (
    <div className="App">
      <h1>Foxhunt Training Monitor</h1>

      {/* Active runs table */}
      <table>
        <thead>
          <tr>
            <th>Run ID</th>
            <th>Model</th>
            <th>Status</th>
            <th>Cost</th>
            <th>Elapsed</th>
          </tr>
        </thead>
        <tbody>
          {runs.map(run => (
            <tr key={run.run_id}>
              <td>{run.run_id}</td>
              <td>{run.model_type}</td>
              <td>{run.status}</td>
              <td>${run.total_cost}</td>
              <td>{run.elapsed_time}</td>
            </tr>
          ))}
        </tbody>
      </table>

      {/* Live metrics chart */}
      <LineChart width={800} height={400} data={metrics}>
        <CartesianGrid strokeDasharray="3 3" />
        <XAxis dataKey="timestamp" />
        <YAxis />
        <Tooltip />
        <Legend />
        <Line type="monotone" dataKey="loss" stroke="#8884d8" />
        <Line type="monotone" dataKey="q_buy" stroke="#82ca9d" />
      </LineChart>
    </div>
  );
}

export default App;

Estimated Effort

• Database setup + ORM models: 1 day
• Backend API + WebSocket: 2-3 days
• Frontend components: 2-3 days
• Integration + testing: 2 days
• Total: 7-9 days (1-2 weeks)

Recommendation: Defer to Phase 2 (Priorities 1-3 provide 80% of value).

---

Implementation Roadmap

Phase 1: Quick Wins (1 Week)

Week 1:

• Day 1: Fix Rust CLI (Priority 1)
• Days 2-3: Enhanced Python metrics (Priority 2, Part 1)
• Days 4-5: Cost tracking + Terminal UI (Priority 2, Part 2)

Deliverables:

• ✅ Working Rust CLI with nested path support
• ✅ Python script with live metrics dashboard
• ✅ Real-time cost tracking with ETA

Value: 60% of total value, 20% of total effort

---

Phase 2: Cost Optimization (1 Week)

Week 2:

• Days 1-3: Alert system (Priority 3, Part 1)
• Days 4-5: Auto-termination (Priority 3, Part 2)

Deliverables:

• ✅ Error/OOM alert system
• ✅ Discord/Slack integration
• ✅ Auto-termination with cost savings

Value: 20% of total value, 30% of total effort

Expected Cost Savings: 20-50% reduction in GPU costs (auto-termination prevents "forgotten pods")

---

Phase 3: Web Dashboard (OPTIONAL - 2 Weeks)

Weeks 3-4:

• Week 3: Backend + database
• Week 4: Frontend + integration

Deliverables:

• ✅ Web-based multi-pod monitoring
• ✅ Historical metrics database
• ✅ Cost analytics and charts

Value: 20% of total value, 50% of total effort

Recommendation: Only pursue if Phases 1-2 are highly successful and there's user demand.

---

Comparison: Polling vs Alternative Approaches

Option A: S3 Polling (RECOMMENDED - Current Approach)

Pros:

• ✅ Simple infrastructure (no webhooks)
• ✅ Works with RunPod S3
• ✅ Can monitor terminated pods
• ✅ Byte-range efficiency (99.9% data savings)
• ✅ 5-10s latency acceptable

Cons:

• ❌ Slight overhead (repeated requests)
• ❌ Not truly real-time (5-10s delay)

Verdict: Optimal for this use case

---

Option B: S3 Event Notifications (NOT VIABLE)

Pros:

• ✅ True real-time (sub-second)
• ✅ Event-driven (no polling)

Cons:

• ❌ RunPod S3 doesn't support S3 events
• ❌ Requires AWS Lambda or webhook endpoint
• ❌ More complex error handling

Verdict: Not possible with RunPod S3

---

Option C: RunPod Logs API (NOT RECOMMENDED)

Pros:

• ✅ Official API
• ✅ Real-time logs

Cons:

• ❌ Only works while pod is running
• ❌ Rate limits
• ❌ Higher latency
• ❌ Pod restart clears logs

Verdict: Worse than S3 polling

---

Cost-Benefit Analysis

Estimated ROI by Priority

Priority	Effort	Value	ROI	Notes
P1: Fix Rust CLI	2-4 hours	High	10x	Unblocks monitoring, minimal effort
P2: Enhanced Python	1-2 days	Very High	5x	Metrics + cost tracking + UI
P3: Alert System	3-5 days	Medium	3x	Cost savings from auto-termination
P4: Web Dashboard	1-2 weeks	Low	1x	Nice-to-have, high effort

Total Effort (P1-P3): 2-3 weeks Total Value: 80% of benefits

Recommendation: Focus on Priorities 1-3. Defer Priority 4 unless there's strong user demand.

---

Success Metrics

Priority 1 Success Criteria

• ✅ Rust CLI can monitor runs with nested S3 paths
• ✅ --run-id parameter works correctly
• ✅ Backward compatibility maintained (pod-id still works)
• ✅ Zero regressions in existing functionality

Priority 2 Success Criteria

• ✅ Metrics extracted correctly (epoch, loss, Q-values, etc.)
• ✅ Cost tracking displays live updates
• ✅ ETA calculation accurate within 10%
• ✅ Terminal UI renders smoothly (no flickering)
• ✅ User can monitor training without checking raw logs

Priority 3 Success Criteria

• ✅ Alerts trigger within 10 seconds of error
• ✅ Discord/Slack notifications delivered reliably
• ✅ Auto-termination saves >20% GPU costs
• ✅ Zero false positives (no accidental terminations)
• ✅ User can set custom cost budgets

Priority 4 Success Criteria (Optional)

• ✅ Web dashboard supports 5+ concurrent runs
• ✅ Real-time updates within 5 seconds
• ✅ Historical metrics queryable (30+ days)
• ✅ Multi-user support (authentication)

---

Risks and Mitigations

Risk 1: Rust CLI Complexity

Risk: Recursive S3 search may be slow or complex Mitigation: Implement Solution A (run-id parameter) with targeted search Fallback: Solution B (full recursive search)

Risk 2: Python Script Dependencies

Risk: Users forget to activate .venv Mitigation: Add clear error messages with setup instructions Fallback: Package as standalone binary with PyInstaller

Risk 3: Alert Fatigue

Risk: Too many alerts overwhelm users Mitigation: Implement severity levels (only send CRITICAL to Discord/Slack) Fallback: Add alert suppression logic (max 1 per 5 minutes)

Risk 4: Auto-Termination Bugs

Risk: Accidental termination of healthy pods Mitigation: Require user confirmation before terminating Fallback: Dry-run mode by default, opt-in for auto-termination

Risk 5: Web Dashboard Scope Creep

Risk: Priority 4 takes too long, delays other work Mitigation: Defer Priority 4 unless Priorities 1-3 succeed Fallback: Use simple Terminal UI instead of web dashboard

---

Conclusion

This design provides a clear roadmap for enhancing the Foxhunt monitoring system with 4 priority tiers balancing quick wins and long-term improvements.

Key Recommendations:

1. Implement Priority 1 immediately (2-4 hours) - Unblocks Rust CLI monitoring
2. Implement Priority 2 next (1-2 days) - Provides 60% of total value
3. Implement Priority 3 if budget allows (3-5 days) - Saves 20-50% GPU costs
4. Defer Priority 4 (1-2 weeks) - Only if strong user demand

Expected Outcomes:

• ✅ Real-time monitoring with live metrics
• ✅ Cost tracking and auto-termination
• ✅ 20-50% reduction in GPU costs
• ✅ Better UX for training runs

Total ROI: 5-10x improvement in monitoring capabilities with 2-3 weeks of effort (Priorities 1-3).

Next Steps: Proceed to implementation with Priority 1 quick fix.