foxhunt/TUNING_PIPELINE_INSTRUCTIONS.md

Hyperparameter Tuning Pipeline - Monitoring Instructions

Pipeline Status: ✅ ACTIVE Started: 2025-10-14 16:57 Expected Completion: 2025-10-15 08:13 Total Duration: ~13.7 hours

---

🎯 Quick Status Check

Run this command anytime to see current status:

/home/jgrusewski/Work/foxhunt/scripts/quick_status.sh

Current Progress (as of 18:00):

• ✅ Auto-monitor: RUNNING (PID 3991060)
• ⏳ DQN: 20/50 trials (40%), Runtime: 1h 3m, ETA: 19:28
• ⏳ PPO: Waiting for DQN
• ⏳ TFT: Waiting for PPO
• ⏳ MAMBA-2: Waiting for TFT
• ⏳ Liquid: Waiting for MAMBA-2

---

📊 Monitoring Commands

Real-Time Dashboard (Recommended)

# Auto-updating dashboard (refreshes every 30 seconds)
watch -n 30 /home/jgrusewski/Work/foxhunt/scripts/dashboard_monitor.sh

# Or run dashboard once
/home/jgrusewski/Work/foxhunt/scripts/dashboard_monitor.sh

Individual Model Logs

# DQN (currently running)
tail -f /tmp/tuning_run.log

# PPO (starts when DQN completes)
tail -f /tmp/ppo_tuning_run.log

# TFT (starts when PPO completes)
tail -f /tmp/tft_tuning_run.log

# MAMBA-2 (starts when TFT completes)
tail -f /tmp/mamba2_tuning_run.log

# Liquid (starts when MAMBA-2 completes)
tail -f /tmp/liquid_tuning_run.log

Pipeline Status Files

# Overall pipeline status
cat /tmp/tuning_pipeline_status.txt

# Auto-monitor log
tail -f /tmp/auto_monitor.log

# Sequential launcher log (after DQN completes)
tail -f /tmp/sequential_tuning.log

GPU Monitoring

# Live GPU status (updates every 5 seconds)
nvidia-smi -l 5

# One-time GPU check
nvidia-smi

# GPU with specific metrics
nvidia-smi --query-gpu=index,name,utilization.gpu,memory.used,memory.total,temperature.gpu --format=csv

Process Monitoring

# Check all tuning processes
ps aux | grep tune_hyperparameters

# Check specific model PIDs
cat /tmp/dqn_tuning.pid       # DQN (3911478)
cat /tmp/ppo_tuning.pid       # PPO (when started)
cat /tmp/tft_tuning.pid       # TFT (when started)
cat /tmp/mamba2_tuning.pid    # MAMBA-2 (when started)
cat /tmp/liquid_tuning.pid    # Liquid (when started)

# Check auto-monitor PID
cat /tmp/auto_monitor.pid     # (3991060)
ps -p $(cat /tmp/auto_monitor.pid) -o etime,pid,cmd

---

⏱️ Expected Timeline

Model	Start Time	Duration	End Time	Status
DQN	16:57	~2.5h	~19:28	⏳ 40% (20/50 trials)
PPO	19:28	~3.2h	~22:42	⏳ Pending
TFT	22:42	~4.2h	~02:54	⏳ Pending
MAMBA-2	02:54	~2.1h	~05:00	⏳ Pending
Liquid	05:00	~1.7h	~06:42	⏳ Pending

Note: Times are approximate and may vary by ±20% based on convergence speed.

---

🚨 What to Watch For

Normal Operation Indicators

• ✅ GPU utilization: 30-60%
• ✅ GPU memory: <2048 MiB (under 50% of 4096 MiB)
• ✅ GPU temperature: <85°C
• ✅ Trial completion: Every ~3-4 minutes
• ✅ Sharpe ratios: 1.5-3.0 range
• ✅ Loss decreasing: <0.1 typically

Warning Signs

• ⚠️ GPU utilization: >90% sustained (possible deadlock)
• ⚠️ GPU memory: >3500 MiB (risk of OOM)
• ⚠️ GPU temperature: >85°C (thermal throttling)
• ⚠️ No trial completion: >10 minutes (possible hang)
• ⚠️ Sharpe ratios: <0.5 (poor hyperparameters)

Error Conditions

• ❌ CUDA Out of Memory (OOM)
• ❌ Process crashed (no PID in ps aux)
• ❌ Auto-monitor stopped
• ❌ GPU temperature: >95°C (emergency)

---

🔧 Troubleshooting

If DQN or Any Model Hangs

# Check if process is still alive
ps -p 3911478  # Replace with actual PID

# Check GPU status
nvidia-smi

# Check last log entries
tail -50 /tmp/tuning_run.log  # Or appropriate model log

# If truly hung (no progress for >15 minutes), kill and restart
kill -9 3911478  # Replace with actual PID

# Restart DQN manually
nohup /home/jgrusewski/Work/foxhunt/target/release/examples/tune_hyperparameters \
  --model DQN \
  --num-trials 50 \
  --epochs-per-trial 50 \
  --data-dir test_data/real/databento/ml_training \
  --output results/dqn_tuning_50trials.json \
  > /tmp/tuning_run.log 2>&1 &

echo $! > /tmp/dqn_tuning.pid

If CUDA Out of Memory (OOM) Occurs

# 1. Note which model failed
grep -i "out of memory\|OOM" /tmp/*tuning*.log

# 2. Kill the failed process
kill -9 $(cat /tmp/<model>_tuning.pid)

# 3. Edit tuning_config.yaml to reduce batch size
nano config/tuning_config.yaml

# Recommended batch size reductions:
# DQN: 256 → 128
# PPO: 256 → 128
# TFT: 128 → 64
# MAMBA-2: 256 → 128
# Liquid: 256 → 128

# 4. Restart the failed model
nohup /home/jgrusewski/Work/foxhunt/target/release/examples/tune_hyperparameters \
  --model <MODEL> \
  --num-trials 50 \
  --epochs-per-trial 50 \
  --data-dir test_data/real/databento/ml_training \
  --output results/<model>_tuning_50trials.json \
  > /tmp/<model>_tuning_run.log 2>&1 &

echo $! > /tmp/<model>_tuning.pid

If Auto-Monitor Stops

# Check if it's still running
ps aux | grep auto_monitor_and_launch.sh

# If not, restart it
nohup /home/jgrusewski/Work/foxhunt/scripts/auto_monitor_and_launch.sh \
  > /tmp/auto_monitor.log 2>&1 &

echo $! > /tmp/auto_monitor.pid

If Sequential Launcher Doesn't Start

# Check if DQN actually completed
ps -p 3911478  # Should return "no such process"
grep -c "Trial .* completed" /tmp/tuning_run.log  # Should be 50

# Manually launch sequential tuner
nohup /home/jgrusewski/Work/foxhunt/scripts/sequential_tuning_launcher.sh \
  > /tmp/sequential_tuning.log 2>&1 &

echo $! > /tmp/sequential_launcher.pid

---

📈 Results Extraction

When All Models Complete

# Extract best hyperparameters
cd /home/jgrusewski/Work/foxhunt
python3 scripts/extract_best_hyperparameters.py

# This will:
# 1. Parse all JSON result files in results/
# 2. Find best trial for each model (by Sharpe ratio)
# 3. Extract optimal hyperparameters
# 4. Update HYPERPARAMETER_TUNING_EXECUTION_REPORT.md

# View updated report
cat HYPERPARAMETER_TUNING_EXECUTION_REPORT.md

Manual Results Inspection

# Check if result files exist
ls -lh results/*_tuning_50trials.json

# View raw JSON (pretty-printed)
jq . results/dqn_tuning_50trials.json | less

# Find best trial manually
jq '[.trials[] | select(.status=="completed")] | max_by(.sharpe_ratio)' \
  results/dqn_tuning_50trials.json

# Extract specific hyperparameter
jq '[.trials[] | select(.status=="completed")] | max_by(.sharpe_ratio) | .hyperparameters.learning_rate' \
  results/dqn_tuning_50trials.json

---

📁 Important Files & Locations

Scripts

• /home/jgrusewski/Work/foxhunt/scripts/auto_monitor_and_launch.sh - Auto-monitor
• /home/jgrusewski/Work/foxhunt/scripts/sequential_tuning_launcher.sh - Sequential launcher
• /home/jgrusewski/Work/foxhunt/scripts/dashboard_monitor.sh - Dashboard
• /home/jgrusewski/Work/foxhunt/scripts/quick_status.sh - Quick status
• /home/jgrusewski/Work/foxhunt/scripts/extract_best_hyperparameters.py - Results extractor

Logs

• /tmp/tuning_run.log - DQN log
• /tmp/ppo_tuning_run.log - PPO log
• /tmp/tft_tuning_run.log - TFT log
• /tmp/mamba2_tuning_run.log - MAMBA-2 log
• /tmp/liquid_tuning_run.log - Liquid log
• /tmp/auto_monitor.log - Auto-monitor log
• /tmp/sequential_tuning.log - Sequential launcher log
• /tmp/tuning_pipeline_status.txt - Pipeline status file

PIDs

• /tmp/dqn_tuning.pid - DQN PID (3911478)
• /tmp/ppo_tuning.pid - PPO PID (when started)
• /tmp/tft_tuning.pid - TFT PID (when started)
• /tmp/mamba2_tuning.pid - MAMBA-2 PID (when started)
• /tmp/liquid_tuning.pid - Liquid PID (when started)
• /tmp/auto_monitor.pid - Auto-monitor PID (3991060)
• /tmp/sequential_launcher.pid - Sequential launcher PID (when started)

Results

• /home/jgrusewski/Work/foxhunt/results/dqn_tuning_50trials.json - DQN results
• /home/jgrusewski/Work/foxhunt/results/ppo_tuning_50trials.json - PPO results
• /home/jgrusewski/Work/foxhunt/results/tft_tuning_50trials.json - TFT results
• /home/jgrusewski/Work/foxhunt/results/mamba2_tuning_50trials.json - MAMBA-2 results
• /home/jgrusewski/Work/foxhunt/results/liquid_tuning_50trials.json - Liquid results

Reports

• /home/jgrusewski/Work/foxhunt/HYPERPARAMETER_TUNING_EXECUTION_REPORT.md - Main report
• /home/jgrusewski/Work/foxhunt/TUNING_PIPELINE_INSTRUCTIONS.md - This file

---

🔔 Monitoring Schedule (Recommended)

Every 30 Minutes

# Quick status check
/home/jgrusewski/Work/foxhunt/scripts/quick_status.sh

# Or watch dashboard
watch -n 30 /home/jgrusewski/Work/foxhunt/scripts/dashboard_monitor.sh

Before Going to Sleep (18:00-20:00)

• ✅ Verify DQN is progressing (should be ~30-40 trials by 20:00)
• ✅ Check GPU temperature (<85°C)
• ✅ Verify auto-monitor is running
• ✅ Check no OOM errors in log

Morning Check (06:00-08:00)

• ✅ Verify all models completed or check which is running
• ✅ Check for any errors in logs
• ✅ Run results extraction if complete

When Pipeline Completes (ETA 08:13)

# 1. Extract results
python3 scripts/extract_best_hyperparameters.py

# 2. Review report
cat HYPERPARAMETER_TUNING_EXECUTION_REPORT.md

# 3. Check all result files exist
ls -lh results/*_tuning_50trials.json

# 4. Verify 50 trials per model
for model in dqn ppo tft mamba2 liquid; do
  echo "$model: $(jq '[.trials[] | select(.status=="completed")] | length' results/${model}_tuning_50trials.json) trials"
done

---

📞 Emergency Contacts & Escalation

If System Becomes Unresponsive

# Check system load
uptime

# Check disk space
df -h

# Check memory
free -h

# Kill all tuning processes if necessary (LAST RESORT)
pkill -9 -f tune_hyperparameters

If Multiple OOM Errors Occur

This indicates insufficient GPU memory. Options:

1. Reduce batch sizes more aggressively (32 → 16 → 8)
2. Reduce epochs per trial (50 → 25)
3. Use CPU instead of GPU (much slower, not recommended)

If Temperature Exceeds 95°C

# Emergency shutdown of all tuning
pkill -9 -f tune_hyperparameters

# Let GPU cool down (wait 10-15 minutes)
watch -n 5 nvidia-smi

# Check laptop cooling/vents
# Consider using cooling pad
# Reduce room temperature if possible

---

✅ Success Criteria

Pipeline is successful when:

• ✅ All 5 models complete 50 trials (250 total)
• ✅ All result files exist and are valid JSON
• ✅ Best Sharpe ratios extracted for each model
• ✅ Hyperparameters show variation across trials
• ✅ No OOM or thermal errors occurred
• ✅ Training times within expected ranges

---

📊 Expected Outcomes

DQN

• Best Sharpe: 2.0-3.5
• Loss: 0.01-0.05
• Learning rate: 1e-4 to 5e-4 (likely)
• Batch size: 64-128 (likely)

PPO

• Best Sharpe: 2.5-4.0
• Loss: 0.02-0.08
• Learning rate: 3e-4 to 7e-4 (likely)
• Clip epsilon: 0.15-0.25 (likely)

TFT

• Best Sharpe: 2.0-3.0
• Loss: 0.03-0.10
• Learning rate: 5e-5 to 2e-4 (likely)
• Hidden size: 128-192 (likely)

MAMBA-2

• Best Sharpe: 2.5-4.0
• Loss: 0.02-0.06
• Learning rate: 1e-4 to 5e-4 (likely)
• State size: 32-48 (likely)

Liquid

• Best Sharpe: 2.0-3.5
• Loss: 0.02-0.07
• Learning rate: 2e-4 to 6e-4 (likely)
• ODE solver steps: 5-8 (likely)

---

Last Updated: 2025-10-14 18:00 Next Update: Check every 30 minutes or when models complete