281 lines
8.9 KiB
Markdown
281 lines
8.9 KiB
Markdown
# Monitor Logs Quick Reference
|
|
|
|
**Script**: `scripts/monitor_logs.py`
|
|
**Purpose**: Real-time S3 log monitoring for RunPod ML training runs
|
|
|
|
---
|
|
|
|
## Requirements
|
|
|
|
1. **Virtual Environment**: MUST activate `.venv` first
|
|
```bash
|
|
source .venv/bin/activate
|
|
```
|
|
|
|
2. **Dependencies**: Already installed in `.venv`
|
|
- `rich` (colored output)
|
|
- `boto3` (S3 access)
|
|
- `pydantic-settings` (config)
|
|
|
|
3. **Credentials**: `.env.runpod` with S3 credentials
|
|
|
|
---
|
|
|
|
## Usage Examples
|
|
|
|
### 1. List Recent Runs
|
|
```bash
|
|
# Default: Show 20 most recent runs
|
|
python3 scripts/monitor_logs.py
|
|
|
|
# Custom limit
|
|
python3 scripts/monitor_logs.py --limit 10
|
|
```
|
|
|
|
**Output**: Formatted table with:
|
|
- Run ID
|
|
- Model type (MAMBA2, DQN, PPO, TFT)
|
|
- Last modified time (relative)
|
|
- Log file size
|
|
- Hyperopt indicator (✓ if `trials.json` exists)
|
|
|
|
### 2. Monitor Specific Run (One-time Snapshot)
|
|
```bash
|
|
# View current log contents (no streaming)
|
|
python3 scripts/monitor_logs.py --run-id run_20251029_145151_hyperopt
|
|
```
|
|
|
|
### 3. Monitor with Continuous Streaming
|
|
```bash
|
|
# Stream logs in real-time until completion
|
|
python3 scripts/monitor_logs.py --run-id run_20251029_145151_hyperopt --follow
|
|
```
|
|
|
|
**Features**:
|
|
- Tail new log entries every 5 seconds (configurable)
|
|
- Color-coded output:
|
|
- **Red**: Errors (CUDA OOM, RuntimeError, etc.)
|
|
- **Yellow**: Warnings
|
|
- **Green**: Success messages, completion
|
|
- Auto-detects training completion
|
|
- Shows hyperopt trial count every 30 seconds
|
|
|
|
### 4. Monitor with Timeout
|
|
```bash
|
|
# Stream for maximum 30 minutes
|
|
python3 scripts/monitor_logs.py --run-id run_20251029_145151_hyperopt --follow --timeout 30m
|
|
|
|
# Stream for 2 hours
|
|
python3 scripts/monitor_logs.py --run-id run_20251029_145151_hyperopt --follow --timeout 2h
|
|
|
|
# Stream for 45 seconds
|
|
python3 scripts/monitor_logs.py --run-id run_20251029_145151_hyperopt --follow --timeout 45s
|
|
```
|
|
|
|
**Timeout Formats**:
|
|
- `30s` = 30 seconds
|
|
- `15m` = 15 minutes
|
|
- `2h` = 2 hours
|
|
|
|
### 5. Custom Poll Interval
|
|
```bash
|
|
# Check for new logs every 10 seconds (default: 5s)
|
|
python3 scripts/monitor_logs.py --run-id run_20251029_145151_hyperopt --follow --interval 10
|
|
```
|
|
|
|
### 6. Monitor by Pod ID (Alternative)
|
|
```bash
|
|
# Monitor using pod ID instead of run ID
|
|
python3 scripts/monitor_logs.py --pod-id w4srx0tgm5hfgu --follow
|
|
```
|
|
|
|
---
|
|
|
|
## S3 Structure
|
|
|
|
```
|
|
s3://se3zdnb5o4/
|
|
├── ml_training/
|
|
│ └── training_runs/
|
|
│ ├── mamba2/
|
|
│ │ └── run_20251029_145151_hyperopt/
|
|
│ │ ├── logs/
|
|
│ │ │ └── training.log # Tailed by script
|
|
│ │ ├── hyperopt/
|
|
│ │ │ └── trials.json # Checked every 30s
|
|
│ │ └── models/
|
|
│ │ └── best_model.safetensors
|
|
│ ├── dqn/
|
|
│ ├── ppo/
|
|
│ └── tft/
|
|
```
|
|
|
|
---
|
|
|
|
## Output Examples
|
|
|
|
### Listing Runs
|
|
```
|
|
Recent Training Runs
|
|
╭──────────────────────────────┬────────┬───────────────┬──────────┬────────╮
|
|
│ Run ID │ Model │ Last Modified │ Log Size │ Trials │
|
|
├──────────────────────────────┼────────┼───────────────┼──────────┼────────┤
|
|
│ run_20251029_223100_hyperopt │ MAMBA2 │ 39m ago │ 1.0KB │ ✓ │
|
|
│ run_20251029_222832_hyperopt │ MAMBA2 │ 51m ago │ 471B │ - │
|
|
│ run_20251029_191027_hyperopt │ MAMBA2 │ 4h ago │ 468B │ - │
|
|
╰──────────────────────────────┴────────┴───────────────┴──────────┴────────╯
|
|
```
|
|
|
|
### Streaming Logs
|
|
```
|
|
╭──────────────────────────────── Log Monitor ─────────────────────────────────╮
|
|
│ Monitoring Run: run_20251029_145151_hyperopt │
|
|
│ Model: MAMBA2 │
|
|
│ Log: ml_training/training_runs/mamba2/run_20251029_145151_hyperopt/logs/... │
|
|
│ Hyperopt: Yes │
|
|
╰──────────────────────────────────────────────────────────────────────────────╯
|
|
|
|
Streaming logs... (Press Ctrl+C to stop)
|
|
──────────────────────────────────────────────────────────────────────
|
|
[2025-10-29 15:08:07] === Starting MAMBA-2 Trial ===
|
|
Params: Mamba2Params { ... }
|
|
[2025-10-29 15:10:41] Training completed in 154.23s: val_loss=0.089249
|
|
📊 Hyperopt trials: 3
|
|
──────────────────────────────────────────────────────────────────────
|
|
✅ Training completed!
|
|
```
|
|
|
|
---
|
|
|
|
## Error Patterns (Auto-Detected)
|
|
|
|
Script highlights these patterns in **RED**:
|
|
- `CUDA out of memory`
|
|
- `RuntimeError:`
|
|
- `AssertionError:`
|
|
- `FAILED:`
|
|
- `ERROR:`
|
|
- `panic!`
|
|
|
|
Script highlights these patterns in **GREEN**:
|
|
- `Training complete`
|
|
- `Model saved to`
|
|
- `✓ Training finished`
|
|
- `SUCCESS:`
|
|
- `Hyperparameter optimization complete`
|
|
|
|
---
|
|
|
|
## Comparison: monitor_logs.py vs monitor_hyperopt.sh
|
|
|
|
| Feature | monitor_logs.py | monitor_hyperopt.sh |
|
|
|---------|----------------|---------------------|
|
|
| **List runs** | ✅ Formatted table | ❌ Manual S3 listing |
|
|
| **Real-time streaming** | ✅ Byte-range tailing | ❌ Periodic polling |
|
|
| **Color output** | ✅ Rich colors | ⚠️ Basic ANSI |
|
|
| **Auto-completion detection** | ✅ Pattern matching | ❌ Manual check |
|
|
| **Timeout support** | ✅ Configurable | ❌ None |
|
|
| **Hyperopt trials** | ✅ Auto-detected | ⚠️ Manual check |
|
|
| **Pod monitoring** | ✅ Via PodMonitor | ❌ SSH only |
|
|
|
|
**Recommendation**: Use `monitor_logs.py` for all log monitoring tasks.
|
|
|
|
---
|
|
|
|
## Troubleshooting
|
|
|
|
### Error: "Not running in a virtual environment"
|
|
```bash
|
|
# Solution: Activate .venv
|
|
source .venv/bin/activate
|
|
```
|
|
|
|
### Error: "Configuration file not found"
|
|
```bash
|
|
# Solution: Ensure .env.runpod exists in project root
|
|
ls -la .env.runpod
|
|
|
|
# If missing, copy from template or restore from backup
|
|
```
|
|
|
|
### Error: "Run not found"
|
|
```bash
|
|
# Solution: List available runs first
|
|
python3 scripts/monitor_logs.py
|
|
|
|
# Verify run ID exactly matches (case-sensitive)
|
|
```
|
|
|
|
### No new logs appearing
|
|
- Check if training is actually running (pod status)
|
|
- Verify S3 sync is working (check S3 directly)
|
|
- Increase poll interval if network is slow (`--interval 10`)
|
|
|
|
---
|
|
|
|
## Integration with Deployment
|
|
|
|
### Standard Workflow
|
|
```bash
|
|
# 1. Deploy training pod
|
|
python3 scripts/runpod_deploy.py --gpu-type "RTX A4000" --monitor
|
|
|
|
# 2. Monitor logs separately (in another terminal)
|
|
source .venv/bin/activate
|
|
python3 scripts/monitor_logs.py --follow --timeout 2h
|
|
|
|
# 3. List all runs to find completed ones
|
|
python3 scripts/monitor_logs.py --limit 50
|
|
```
|
|
|
|
### Automated Monitoring
|
|
```bash
|
|
# Deploy with auto-monitoring (blocks until complete)
|
|
python3 scripts/runpod_deploy.py --monitor --auto-stop --timeout 2h
|
|
```
|
|
|
|
---
|
|
|
|
## Architecture
|
|
|
|
### Log Tailing Method
|
|
- **NO SSH required**: Pure S3 byte-range requests
|
|
- **Efficient**: Only fetches new bytes since last check
|
|
- **Real-time**: 5-second polling (configurable)
|
|
- **Robust**: Handles missing files, network errors
|
|
|
|
### Classes Used
|
|
- `PodMonitor`: Pod status and log streaming (from `foxhunt_runpod`)
|
|
- `S3Client`: S3 operations with retry logic
|
|
- `RunPodConfig`: Configuration from `.env.runpod`
|
|
|
|
### Performance
|
|
- **S3 API calls**: ~12/minute (5s interval)
|
|
- **Bandwidth**: ~1KB-1MB per check (depends on log size)
|
|
- **Latency**: <500ms per S3 request
|
|
- **Cost**: Negligible (S3 GET requests: $0.0004/1000)
|
|
|
|
---
|
|
|
|
## Future Enhancements
|
|
|
|
1. **Web UI**: Real-time dashboard with WebSockets
|
|
2. **Multi-run monitoring**: Monitor multiple runs simultaneously
|
|
3. **Alert notifications**: Slack/email on completion/errors
|
|
4. **Log analytics**: Parse metrics, plot training curves
|
|
5. **Historical replay**: View logs from completed runs with timestamps
|
|
|
|
---
|
|
|
|
## Related Scripts
|
|
|
|
- `scripts/runpod_deploy.py`: Deploy training pods with optional monitoring
|
|
- `scripts/upload_binary.py`: Upload training binaries to S3
|
|
- `scripts/monitor_hyperopt.sh`: Legacy monitoring script (deprecated)
|
|
|
|
---
|
|
|
|
**Last Updated**: 2025-10-30
|
|
**Status**: ✅ Production Ready
|
|
**Tests**: Manual validation complete
|