foxhunt/TUNING_QUICKSTART_GUIDE.md

Hyperparameter Tuning Quickstart Guide

Quick Reference for Managing Multi-Model Hyperparameter Optimization

---

Current Status (Live)

# Check real-time status
/home/jgrusewski/Work/foxhunt/scripts/monitor_tuning.sh

# Or with auto-refresh (every 30 seconds)
watch -n 30 /home/jgrusewski/Work/foxhunt/scripts/monitor_tuning.sh

Current Progress: DQN tuning running (Trial 12/50, 24% complete) ETA: 19:25 CEST (2025-10-14)

---

Quick Commands

Monitor DQN Progress

# Tail log (live updates)
tail -f /tmp/tuning_run.log

# Count completed trials
grep -c "Trial .* completed" /tmp/tuning_run.log

# Check process status
ps aux | grep 3911478

# GPU utilization
nvidia-smi

Launch Next Model (PPO)

# Manual launch (after DQN completes)
/home/jgrusewski/Work/foxhunt/target/release/examples/tune_hyperparameters \
  --model PPO \
  --num-trials 50 \
  --epochs-per-trial 50 \
  --data-dir test_data/real/databento/ml_training \
  --output results/ppo_tuning_50trials.json \
  > /tmp/ppo_tuning_run.log 2>&1 &

echo $! > /tmp/ppo_tuning.pid

# Automatic launch (waits for DQN, then starts PPO)
nohup /home/jgrusewski/Work/foxhunt/scripts/auto_launch_ppo.sh > /tmp/auto_launch_ppo.log 2>&1 &

Sequential Tuning (All Models)

# Launch all 5 models sequentially (12-14 hours total)
nohup /home/jgrusewski/Work/foxhunt/scripts/sequential_tuning_launcher.sh \
  > /tmp/sequential_tuning.log 2>&1 &

---

Results Location

After tuning completes, results will be in:

/home/jgrusewski/Work/foxhunt/results/
├── dqn_tuning_50trials.json      # DQN best hyperparameters
├── ppo_tuning_50trials.json      # PPO best hyperparameters
├── tft_tuning_50trials.json      # TFT best hyperparameters
├── mamba2_tuning_50trials.json   # MAMBA-2 best hyperparameters
└── liquid_tuning_50trials.json   # Liquid best hyperparameters

---

Extract Best Hyperparameters

# View best trial for DQN
jq '.best_trial' results/dqn_tuning_50trials.json

# Extract best learning rate
jq '.best_trial.config.learning_rate' results/dqn_tuning_50trials.json

# Sort all trials by Sharpe ratio
jq '.all_results | sort_by(.sharpe_ratio) | reverse | .[0:5]' results/dqn_tuning_50trials.json

---

Troubleshooting

Process Died or Stopped

# Check if process is running
ps aux | grep tune_hyperparameters

# Restart from checkpoint (if supported)
# Note: Current implementation doesn't support checkpointing
# Need to restart from scratch

GPU Out of Memory (OOM)

# Reduce batch size for TFT/MAMBA-2
# Edit tuning_config.yaml or use command line:

/home/jgrusewski/Work/foxhunt/target/release/examples/tune_hyperparameters \
  --model TFT \
  --num-trials 50 \
  --epochs-per-trial 50 \
  --batch-size 16  # Reduced from 32

Identical Sharpe Ratios (All trials = 2.00)

# This is currently observed for DQN
# Investigation needed after completion

# Possible causes:
# 1. Deterministic random seed
# 2. Evaluation metric not sensitive to hyperparameters
# 3. Model already well-tuned at default settings

# Check if different hyperparameters are being tested:
grep "learning_rate\|batch_size" /tmp/tuning_run.log | head -20

---

Model-Specific Notes

DQN

• Trials: 50
• Duration: 2.5-3 hours
• Memory: ~150MB VRAM (safe for RTX 3050 Ti)
• Status: Running (Trial 12/50)

PPO

• Trials: 50
• Duration: 3-4 hours
• Memory: ~200MB VRAM
• Recommendation: Launch after DQN completes (19:25 CEST)

TFT

• Trials: 50
• Duration: 4-5 hours
• Memory: ~1.8GB VRAM (45% of 4GB)
• Warning: Monitor for OOM, reduce batch size to 16 if needed

MAMBA-2

• Trials: 50
• Duration: 2-3 hours
• Memory: ~2.5GB VRAM (62% of 4GB)
• Warning: Gradient checkpointing enabled, may still OOM with batch_size=16

Liquid

• Trials: 50
• Duration: 1.5-2 hours
• Memory: ~120MB VRAM (safe)
• Note: Fastest model to tune

---

Timeline (Estimated)

Model	Start	End	Duration
DQN	16:57	19:25	2.5h
PPO	19:25	22:45	3.2h
TFT	22:45	02:55	4.2h
MAMBA-2	02:55	05:00	2.1h
Liquid	05:00	06:40	1.7h
TOTAL	16:57	06:40	13.7h

All times in CEST (Central European Summer Time)

---

Production Training (After Tuning)

Once best hyperparameters are identified:

# 1. Update model configs with best hyperparameters
vim ml/src/trainers/dqn.rs  # Update DQNHyperparameters::default()
vim ml/src/trainers/ppo.rs  # Update PpoHyperparameters::default()
# ... etc for other models

# 2. Run production training (4-6 weeks)
cargo run -p ml --example retrain_all_models --release --features cuda

# 3. Validate models
cargo run -p ml --example comprehensive_model_backtest --release

---

Support

Documentation:

• Full status: /home/jgrusewski/Work/foxhunt/HYPERPARAMETER_TUNING_STATUS.md
• This guide: /home/jgrusewski/Work/foxhunt/TUNING_QUICKSTART_GUIDE.md
• Tuning config: /home/jgrusewski/Work/foxhunt/tuning_config.yaml

Scripts:

• Monitor: /home/jgrusewski/Work/foxhunt/scripts/monitor_tuning.sh
• Auto PPO: /home/jgrusewski/Work/foxhunt/scripts/auto_launch_ppo.sh
• Sequential: /home/jgrusewski/Work/foxhunt/scripts/sequential_tuning_launcher.sh

Logs:

• DQN: /tmp/tuning_run.log
• PPO: /tmp/ppo_tuning_run.log
• TFT: /tmp/tft_tuning_run.log
• MAMBA-2: /tmp/mamba2_tuning_run.log
• Liquid: /tmp/liquid_tuning_run.log

---

Last Updated: 2025-10-14 17:33 CEST Agent: Agent 79 Status: DQN 24% complete (Trial 12/50)