# Hyperparameter Tuning Pipeline - Monitoring Instructions

**Pipeline Status**: ✅ **ACTIVE**
**Started**: 2025-10-14 16:57
**Expected Completion**: 2025-10-15 08:13
**Total Duration**: ~13.7 hours

---

## 🎯 Quick Status Check

**Run this command anytime to see current status:**

```bash
/home/jgrusewski/Work/foxhunt/scripts/quick_status.sh
```

**Current Progress** (as of 18:00):
- ✅ Auto-monitor: RUNNING (PID 3991060)
- ⏳ DQN: 20/50 trials (40%), Runtime: 1h 3m, ETA: 19:28
- ⏳ PPO: Waiting for DQN
- ⏳ TFT: Waiting for PPO
- ⏳ MAMBA-2: Waiting for TFT
- ⏳ Liquid: Waiting for MAMBA-2

---

## 📊 Monitoring Commands

### Real-Time Dashboard (Recommended)

```bash
# Auto-updating dashboard (refreshes every 30 seconds)
watch -n 30 /home/jgrusewski/Work/foxhunt/scripts/dashboard_monitor.sh

# Or run dashboard once
/home/jgrusewski/Work/foxhunt/scripts/dashboard_monitor.sh
```

### Individual Model Logs

```bash
# DQN (currently running)
tail -f /tmp/tuning_run.log

# PPO (starts when DQN completes)
tail -f /tmp/ppo_tuning_run.log

# TFT (starts when PPO completes)
tail -f /tmp/tft_tuning_run.log

# MAMBA-2 (starts when TFT completes)
tail -f /tmp/mamba2_tuning_run.log

# Liquid (starts when MAMBA-2 completes)
tail -f /tmp/liquid_tuning_run.log
```

### Pipeline Status Files

```bash
# Overall pipeline status
cat /tmp/tuning_pipeline_status.txt

# Auto-monitor log
tail -f /tmp/auto_monitor.log

# Sequential launcher log (after DQN completes)
tail -f /tmp/sequential_tuning.log
```

### GPU Monitoring

```bash
# Live GPU status (updates every 5 seconds)
nvidia-smi -l 5

# One-time GPU check
nvidia-smi

# GPU with specific metrics
nvidia-smi --query-gpu=index,name,utilization.gpu,memory.used,memory.total,temperature.gpu --format=csv
```

### Process Monitoring

```bash
# Check all tuning processes
ps aux | grep tune_hyperparameters

# Check specific model PIDs
cat /tmp/dqn_tuning.pid       # DQN (3911478)
cat /tmp/ppo_tuning.pid       # PPO (when started)
cat /tmp/tft_tuning.pid       # TFT (when started)
cat /tmp/mamba2_tuning.pid    # MAMBA-2 (when started)
cat /tmp/liquid_tuning.pid    # Liquid (when started)

# Check auto-monitor PID
cat /tmp/auto_monitor.pid     # (3991060)
ps -p $(cat /tmp/auto_monitor.pid) -o etime,pid,cmd
```

---

## ⏱️ Expected Timeline

| Model | Start Time | Duration | End Time | Status |
|-------|------------|----------|----------|--------|
| DQN | 16:57 | ~2.5h | ~19:28 | ⏳ 40% (20/50 trials) |
| PPO | 19:28 | ~3.2h | ~22:42 | ⏳ Pending |
| TFT | 22:42 | ~4.2h | ~02:54 | ⏳ Pending |
| MAMBA-2 | 02:54 | ~2.1h | ~05:00 | ⏳ Pending |
| Liquid | 05:00 | ~1.7h | ~06:42 | ⏳ Pending |

**Note**: Times are approximate and may vary by ±20% based on convergence speed.

---

## 🚨 What to Watch For

### Normal Operation Indicators
- ✅ GPU utilization: 30-60%
- ✅ GPU memory: <2048 MiB (under 50% of 4096 MiB)
- ✅ GPU temperature: <85°C
- ✅ Trial completion: Every ~3-4 minutes
- ✅ Sharpe ratios: 1.5-3.0 range
- ✅ Loss decreasing: <0.1 typically

### Warning Signs
- ⚠️ GPU utilization: >90% sustained (possible deadlock)
- ⚠️ GPU memory: >3500 MiB (risk of OOM)
- ⚠️ GPU temperature: >85°C (thermal throttling)
- ⚠️ No trial completion: >10 minutes (possible hang)
- ⚠️ Sharpe ratios: <0.5 (poor hyperparameters)

### Error Conditions
- ❌ CUDA Out of Memory (OOM)
- ❌ Process crashed (no PID in `ps aux`)
- ❌ Auto-monitor stopped
- ❌ GPU temperature: >95°C (emergency)

---

## 🔧 Troubleshooting

### If DQN or Any Model Hangs

```bash
# Check if process is still alive
ps -p 3911478  # Replace with actual PID

# Check GPU status
nvidia-smi

# Check last log entries
tail -50 /tmp/tuning_run.log  # Or appropriate model log

# If truly hung (no progress for >15 minutes), kill and restart
kill -9 3911478  # Replace with actual PID

# Restart DQN manually
nohup /home/jgrusewski/Work/foxhunt/target/release/examples/tune_hyperparameters \
  --model DQN \
  --num-trials 50 \
  --epochs-per-trial 50 \
  --data-dir test_data/real/databento/ml_training \
  --output results/dqn_tuning_50trials.json \
  > /tmp/tuning_run.log 2>&1 &

echo $! > /tmp/dqn_tuning.pid
```

### If CUDA Out of Memory (OOM) Occurs

```bash
# 1. Note which model failed
grep -i "out of memory\|OOM" /tmp/*tuning*.log

# 2. Kill the failed process
kill -9 $(cat /tmp/<model>_tuning.pid)

# 3. Edit tuning_config.yaml to reduce batch size
nano config/tuning_config.yaml

# Recommended batch size reductions:
# DQN: 256 → 128
# PPO: 256 → 128
# TFT: 128 → 64
# MAMBA-2: 256 → 128
# Liquid: 256 → 128

# 4. Restart the failed model
nohup /home/jgrusewski/Work/foxhunt/target/release/examples/tune_hyperparameters \
  --model <MODEL> \
  --num-trials 50 \
  --epochs-per-trial 50 \
  --data-dir test_data/real/databento/ml_training \
  --output results/<model>_tuning_50trials.json \
  > /tmp/<model>_tuning_run.log 2>&1 &

echo $! > /tmp/<model>_tuning.pid
```

### If Auto-Monitor Stops

```bash
# Check if it's still running
ps aux | grep auto_monitor_and_launch.sh

# If not, restart it
nohup /home/jgrusewski/Work/foxhunt/scripts/auto_monitor_and_launch.sh \
  > /tmp/auto_monitor.log 2>&1 &

echo $! > /tmp/auto_monitor.pid
```

### If Sequential Launcher Doesn't Start

```bash
# Check if DQN actually completed
ps -p 3911478  # Should return "no such process"
grep -c "Trial .* completed" /tmp/tuning_run.log  # Should be 50

# Manually launch sequential tuner
nohup /home/jgrusewski/Work/foxhunt/scripts/sequential_tuning_launcher.sh \
  > /tmp/sequential_tuning.log 2>&1 &

echo $! > /tmp/sequential_launcher.pid
```

---

## 📈 Results Extraction

### When All Models Complete

```bash
# Extract best hyperparameters
cd /home/jgrusewski/Work/foxhunt
python3 scripts/extract_best_hyperparameters.py

# This will:
# 1. Parse all JSON result files in results/
# 2. Find best trial for each model (by Sharpe ratio)
# 3. Extract optimal hyperparameters
# 4. Update HYPERPARAMETER_TUNING_EXECUTION_REPORT.md

# View updated report
cat HYPERPARAMETER_TUNING_EXECUTION_REPORT.md
```

### Manual Results Inspection

```bash
# Check if result files exist
ls -lh results/*_tuning_50trials.json

# View raw JSON (pretty-printed)
jq . results/dqn_tuning_50trials.json | less

# Find best trial manually
jq '[.trials[] | select(.status=="completed")] | max_by(.sharpe_ratio)' \
  results/dqn_tuning_50trials.json

# Extract specific hyperparameter
jq '[.trials[] | select(.status=="completed")] | max_by(.sharpe_ratio) | .hyperparameters.learning_rate' \
  results/dqn_tuning_50trials.json
```

---

## 📁 Important Files & Locations

### Scripts
- `/home/jgrusewski/Work/foxhunt/scripts/auto_monitor_and_launch.sh` - Auto-monitor
- `/home/jgrusewski/Work/foxhunt/scripts/sequential_tuning_launcher.sh` - Sequential launcher
- `/home/jgrusewski/Work/foxhunt/scripts/dashboard_monitor.sh` - Dashboard
- `/home/jgrusewski/Work/foxhunt/scripts/quick_status.sh` - Quick status
- `/home/jgrusewski/Work/foxhunt/scripts/extract_best_hyperparameters.py` - Results extractor

### Logs
- `/tmp/tuning_run.log` - DQN log
- `/tmp/ppo_tuning_run.log` - PPO log
- `/tmp/tft_tuning_run.log` - TFT log
- `/tmp/mamba2_tuning_run.log` - MAMBA-2 log
- `/tmp/liquid_tuning_run.log` - Liquid log
- `/tmp/auto_monitor.log` - Auto-monitor log
- `/tmp/sequential_tuning.log` - Sequential launcher log
- `/tmp/tuning_pipeline_status.txt` - Pipeline status file

### PIDs
- `/tmp/dqn_tuning.pid` - DQN PID (3911478)
- `/tmp/ppo_tuning.pid` - PPO PID (when started)
- `/tmp/tft_tuning.pid` - TFT PID (when started)
- `/tmp/mamba2_tuning.pid` - MAMBA-2 PID (when started)
- `/tmp/liquid_tuning.pid` - Liquid PID (when started)
- `/tmp/auto_monitor.pid` - Auto-monitor PID (3991060)
- `/tmp/sequential_launcher.pid` - Sequential launcher PID (when started)

### Results
- `/home/jgrusewski/Work/foxhunt/results/dqn_tuning_50trials.json` - DQN results
- `/home/jgrusewski/Work/foxhunt/results/ppo_tuning_50trials.json` - PPO results
- `/home/jgrusewski/Work/foxhunt/results/tft_tuning_50trials.json` - TFT results
- `/home/jgrusewski/Work/foxhunt/results/mamba2_tuning_50trials.json` - MAMBA-2 results
- `/home/jgrusewski/Work/foxhunt/results/liquid_tuning_50trials.json` - Liquid results

### Reports
- `/home/jgrusewski/Work/foxhunt/HYPERPARAMETER_TUNING_EXECUTION_REPORT.md` - Main report
- `/home/jgrusewski/Work/foxhunt/TUNING_PIPELINE_INSTRUCTIONS.md` - This file

---

## 🔔 Monitoring Schedule (Recommended)

### Every 30 Minutes
```bash
# Quick status check
/home/jgrusewski/Work/foxhunt/scripts/quick_status.sh

# Or watch dashboard
watch -n 30 /home/jgrusewski/Work/foxhunt/scripts/dashboard_monitor.sh
```

### Before Going to Sleep (18:00-20:00)
- ✅ Verify DQN is progressing (should be ~30-40 trials by 20:00)
- ✅ Check GPU temperature (<85°C)
- ✅ Verify auto-monitor is running
- ✅ Check no OOM errors in log

### Morning Check (06:00-08:00)
- ✅ Verify all models completed or check which is running
- ✅ Check for any errors in logs
- ✅ Run results extraction if complete

### When Pipeline Completes (ETA 08:13)
```bash
# 1. Extract results
python3 scripts/extract_best_hyperparameters.py

# 2. Review report
cat HYPERPARAMETER_TUNING_EXECUTION_REPORT.md

# 3. Check all result files exist
ls -lh results/*_tuning_50trials.json

# 4. Verify 50 trials per model
for model in dqn ppo tft mamba2 liquid; do
  echo "$model: $(jq '[.trials[] | select(.status=="completed")] | length' results/${model}_tuning_50trials.json) trials"
done
```

---

## 📞 Emergency Contacts & Escalation

### If System Becomes Unresponsive
```bash
# Check system load
uptime

# Check disk space
df -h

# Check memory
free -h

# Kill all tuning processes if necessary (LAST RESORT)
pkill -9 -f tune_hyperparameters
```

### If Multiple OOM Errors Occur
This indicates insufficient GPU memory. Options:
1. Reduce batch sizes more aggressively (32 → 16 → 8)
2. Reduce epochs per trial (50 → 25)
3. Use CPU instead of GPU (much slower, not recommended)

### If Temperature Exceeds 95°C
```bash
# Emergency shutdown of all tuning
pkill -9 -f tune_hyperparameters

# Let GPU cool down (wait 10-15 minutes)
watch -n 5 nvidia-smi

# Check laptop cooling/vents
# Consider using cooling pad
# Reduce room temperature if possible
```

---

## ✅ Success Criteria

Pipeline is successful when:
- ✅ All 5 models complete 50 trials (250 total)
- ✅ All result files exist and are valid JSON
- ✅ Best Sharpe ratios extracted for each model
- ✅ Hyperparameters show variation across trials
- ✅ No OOM or thermal errors occurred
- ✅ Training times within expected ranges

---

## 📊 Expected Outcomes

### DQN
- Best Sharpe: 2.0-3.5
- Loss: 0.01-0.05
- Learning rate: 1e-4 to 5e-4 (likely)
- Batch size: 64-128 (likely)

### PPO
- Best Sharpe: 2.5-4.0
- Loss: 0.02-0.08
- Learning rate: 3e-4 to 7e-4 (likely)
- Clip epsilon: 0.15-0.25 (likely)

### TFT
- Best Sharpe: 2.0-3.0
- Loss: 0.03-0.10
- Learning rate: 5e-5 to 2e-4 (likely)
- Hidden size: 128-192 (likely)

### MAMBA-2
- Best Sharpe: 2.5-4.0
- Loss: 0.02-0.06
- Learning rate: 1e-4 to 5e-4 (likely)
- State size: 32-48 (likely)

### Liquid
- Best Sharpe: 2.0-3.5
- Loss: 0.02-0.07
- Learning rate: 2e-4 to 6e-4 (likely)
- ODE solver steps: 5-8 (likely)

---

**Last Updated**: 2025-10-14 18:00
**Next Update**: Check every 30 minutes or when models complete