# Monitor Logs Quick Reference

**Script**: `scripts/monitor_logs.py`
**Purpose**: Real-time S3 log monitoring for RunPod ML training runs

---

## Requirements

1. **Virtual Environment**: MUST activate `.venv` first
   ```bash
   source .venv/bin/activate
   ```

2. **Dependencies**: Already installed in `.venv`
   - `rich` (colored output)
   - `boto3` (S3 access)
   - `pydantic-settings` (config)

3. **Credentials**: `.env.runpod` with S3 credentials

---

## Usage Examples

### 1. List Recent Runs
```bash
# Default: Show 20 most recent runs
python3 scripts/monitor_logs.py

# Custom limit
python3 scripts/monitor_logs.py --limit 10
```

**Output**: Formatted table with:
- Run ID
- Model type (MAMBA2, DQN, PPO, TFT)
- Last modified time (relative)
- Log file size
- Hyperopt indicator (✓ if `trials.json` exists)

### 2. Monitor Specific Run (One-time Snapshot)
```bash
# View current log contents (no streaming)
python3 scripts/monitor_logs.py --run-id run_20251029_145151_hyperopt
```

### 3. Monitor with Continuous Streaming
```bash
# Stream logs in real-time until completion
python3 scripts/monitor_logs.py --run-id run_20251029_145151_hyperopt --follow
```

**Features**:
- Tail new log entries every 5 seconds (configurable)
- Color-coded output:
  - **Red**: Errors (CUDA OOM, RuntimeError, etc.)
  - **Yellow**: Warnings
  - **Green**: Success messages, completion
- Auto-detects training completion
- Shows hyperopt trial count every 30 seconds

### 4. Monitor with Timeout
```bash
# Stream for maximum 30 minutes
python3 scripts/monitor_logs.py --run-id run_20251029_145151_hyperopt --follow --timeout 30m

# Stream for 2 hours
python3 scripts/monitor_logs.py --run-id run_20251029_145151_hyperopt --follow --timeout 2h

# Stream for 45 seconds
python3 scripts/monitor_logs.py --run-id run_20251029_145151_hyperopt --follow --timeout 45s
```

**Timeout Formats**:
- `30s` = 30 seconds
- `15m` = 15 minutes
- `2h` = 2 hours

### 5. Custom Poll Interval
```bash
# Check for new logs every 10 seconds (default: 5s)
python3 scripts/monitor_logs.py --run-id run_20251029_145151_hyperopt --follow --interval 10
```

### 6. Monitor by Pod ID (Alternative)
```bash
# Monitor using pod ID instead of run ID
python3 scripts/monitor_logs.py --pod-id w4srx0tgm5hfgu --follow
```

---

## S3 Structure

```
s3://se3zdnb5o4/
├── ml_training/
│   └── training_runs/
│       ├── mamba2/
│       │   └── run_20251029_145151_hyperopt/
│       │       ├── logs/
│       │       │   └── training.log       # Tailed by script
│       │       ├── hyperopt/
│       │       │   └── trials.json        # Checked every 30s
│       │       └── models/
│       │           └── best_model.safetensors
│       ├── dqn/
│       ├── ppo/
│       └── tft/
```

---

## Output Examples

### Listing Runs
```
                            Recent Training Runs
╭──────────────────────────────┬────────┬───────────────┬──────────┬────────╮
│ Run ID                       │ Model  │ Last Modified │ Log Size │ Trials │
├──────────────────────────────┼────────┼───────────────┼──────────┼────────┤
│ run_20251029_223100_hyperopt │ MAMBA2 │ 39m ago       │    1.0KB │   ✓    │
│ run_20251029_222832_hyperopt │ MAMBA2 │ 51m ago       │     471B │   -    │
│ run_20251029_191027_hyperopt │ MAMBA2 │ 4h ago        │     468B │   -    │
╰──────────────────────────────┴────────┴───────────────┴──────────┴────────╯
```

### Streaming Logs
```
╭──────────────────────────────── Log Monitor ─────────────────────────────────╮
│ Monitoring Run: run_20251029_145151_hyperopt                                 │
│ Model: MAMBA2                                                                │
│ Log: ml_training/training_runs/mamba2/run_20251029_145151_hyperopt/logs/... │
│ Hyperopt: Yes                                                                │
╰──────────────────────────────────────────────────────────────────────────────╯

Streaming logs... (Press Ctrl+C to stop)
──────────────────────────────────────────────────────────────────────
[2025-10-29 15:08:07] === Starting MAMBA-2 Trial ===
Params: Mamba2Params { ... }
[2025-10-29 15:10:41] Training completed in 154.23s: val_loss=0.089249
📊 Hyperopt trials: 3
──────────────────────────────────────────────────────────────────────
✅ Training completed!
```

---

## Error Patterns (Auto-Detected)

Script highlights these patterns in **RED**:
- `CUDA out of memory`
- `RuntimeError:`
- `AssertionError:`
- `FAILED:`
- `ERROR:`
- `panic!`

Script highlights these patterns in **GREEN**:
- `Training complete`
- `Model saved to`
- `✓ Training finished`
- `SUCCESS:`
- `Hyperparameter optimization complete`

---

## Comparison: monitor_logs.py vs monitor_hyperopt.sh

| Feature | monitor_logs.py | monitor_hyperopt.sh |
|---------|----------------|---------------------|
| **List runs** | ✅ Formatted table | ❌ Manual S3 listing |
| **Real-time streaming** | ✅ Byte-range tailing | ❌ Periodic polling |
| **Color output** | ✅ Rich colors | ⚠️ Basic ANSI |
| **Auto-completion detection** | ✅ Pattern matching | ❌ Manual check |
| **Timeout support** | ✅ Configurable | ❌ None |
| **Hyperopt trials** | ✅ Auto-detected | ⚠️ Manual check |
| **Pod monitoring** | ✅ Via PodMonitor | ❌ SSH only |

**Recommendation**: Use `monitor_logs.py` for all log monitoring tasks.

---

## Troubleshooting

### Error: "Not running in a virtual environment"
```bash
# Solution: Activate .venv
source .venv/bin/activate
```

### Error: "Configuration file not found"
```bash
# Solution: Ensure .env.runpod exists in project root
ls -la .env.runpod

# If missing, copy from template or restore from backup
```

### Error: "Run not found"
```bash
# Solution: List available runs first
python3 scripts/monitor_logs.py

# Verify run ID exactly matches (case-sensitive)
```

### No new logs appearing
- Check if training is actually running (pod status)
- Verify S3 sync is working (check S3 directly)
- Increase poll interval if network is slow (`--interval 10`)

---

## Integration with Deployment

### Standard Workflow
```bash
# 1. Deploy training pod
python3 scripts/runpod_deploy.py --gpu-type "RTX A4000" --monitor

# 2. Monitor logs separately (in another terminal)
source .venv/bin/activate
python3 scripts/monitor_logs.py --follow --timeout 2h

# 3. List all runs to find completed ones
python3 scripts/monitor_logs.py --limit 50
```

### Automated Monitoring
```bash
# Deploy with auto-monitoring (blocks until complete)
python3 scripts/runpod_deploy.py --monitor --auto-stop --timeout 2h
```

---

## Architecture

### Log Tailing Method
- **NO SSH required**: Pure S3 byte-range requests
- **Efficient**: Only fetches new bytes since last check
- **Real-time**: 5-second polling (configurable)
- **Robust**: Handles missing files, network errors

### Classes Used
- `PodMonitor`: Pod status and log streaming (from `foxhunt_runpod`)
- `S3Client`: S3 operations with retry logic
- `RunPodConfig`: Configuration from `.env.runpod`

### Performance
- **S3 API calls**: ~12/minute (5s interval)
- **Bandwidth**: ~1KB-1MB per check (depends on log size)
- **Latency**: <500ms per S3 request
- **Cost**: Negligible (S3 GET requests: $0.0004/1000)

---

## Future Enhancements

1. **Web UI**: Real-time dashboard with WebSockets
2. **Multi-run monitoring**: Monitor multiple runs simultaneously
3. **Alert notifications**: Slack/email on completion/errors
4. **Log analytics**: Parse metrics, plot training curves
5. **Historical replay**: View logs from completed runs with timestamps

---

## Related Scripts

- `scripts/runpod_deploy.py`: Deploy training pods with optional monitoring
- `scripts/upload_binary.py`: Upload training binaries to S3
- `scripts/monitor_hyperopt.sh`: Legacy monitoring script (deprecated)

---

**Last Updated**: 2025-10-30
**Status**: ✅ Production Ready
**Tests**: Manual validation complete