# Hyperparameter Tuning Quickstart Guide
**Quick Reference for Managing Multi-Model Hyperparameter Optimization**

---

## Current Status (Live)

```bash
# Check real-time status
/home/jgrusewski/Work/foxhunt/scripts/monitor_tuning.sh

# Or with auto-refresh (every 30 seconds)
watch -n 30 /home/jgrusewski/Work/foxhunt/scripts/monitor_tuning.sh
```

**Current Progress**: DQN tuning running (Trial 12/50, 24% complete)
**ETA**: 19:25 CEST (2025-10-14)

---

## Quick Commands

### Monitor DQN Progress
```bash
# Tail log (live updates)
tail -f /tmp/tuning_run.log

# Count completed trials
grep -c "Trial .* completed" /tmp/tuning_run.log

# Check process status
ps aux | grep 3911478

# GPU utilization
nvidia-smi
```

### Launch Next Model (PPO)
```bash
# Manual launch (after DQN completes)
/home/jgrusewski/Work/foxhunt/target/release/examples/tune_hyperparameters \
  --model PPO \
  --num-trials 50 \
  --epochs-per-trial 50 \
  --data-dir test_data/real/databento/ml_training \
  --output results/ppo_tuning_50trials.json \
  > /tmp/ppo_tuning_run.log 2>&1 &

echo $! > /tmp/ppo_tuning.pid

# Automatic launch (waits for DQN, then starts PPO)
nohup /home/jgrusewski/Work/foxhunt/scripts/auto_launch_ppo.sh > /tmp/auto_launch_ppo.log 2>&1 &
```

### Sequential Tuning (All Models)
```bash
# Launch all 5 models sequentially (12-14 hours total)
nohup /home/jgrusewski/Work/foxhunt/scripts/sequential_tuning_launcher.sh \
  > /tmp/sequential_tuning.log 2>&1 &
```

---

## Results Location

After tuning completes, results will be in:
```
/home/jgrusewski/Work/foxhunt/results/
├── dqn_tuning_50trials.json      # DQN best hyperparameters
├── ppo_tuning_50trials.json      # PPO best hyperparameters
├── tft_tuning_50trials.json      # TFT best hyperparameters
├── mamba2_tuning_50trials.json   # MAMBA-2 best hyperparameters
└── liquid_tuning_50trials.json   # Liquid best hyperparameters
```

---

## Extract Best Hyperparameters

```bash
# View best trial for DQN
jq '.best_trial' results/dqn_tuning_50trials.json

# Extract best learning rate
jq '.best_trial.config.learning_rate' results/dqn_tuning_50trials.json

# Sort all trials by Sharpe ratio
jq '.all_results | sort_by(.sharpe_ratio) | reverse | .[0:5]' results/dqn_tuning_50trials.json
```

---

## Troubleshooting

### Process Died or Stopped
```bash
# Check if process is running
ps aux | grep tune_hyperparameters

# Restart from checkpoint (if supported)
# Note: Current implementation doesn't support checkpointing
# Need to restart from scratch
```

### GPU Out of Memory (OOM)
```bash
# Reduce batch size for TFT/MAMBA-2
# Edit tuning_config.yaml or use command line:

/home/jgrusewski/Work/foxhunt/target/release/examples/tune_hyperparameters \
  --model TFT \
  --num-trials 50 \
  --epochs-per-trial 50 \
  --batch-size 16  # Reduced from 32
```

### Identical Sharpe Ratios (All trials = 2.00)
```bash
# This is currently observed for DQN
# Investigation needed after completion

# Possible causes:
# 1. Deterministic random seed
# 2. Evaluation metric not sensitive to hyperparameters
# 3. Model already well-tuned at default settings

# Check if different hyperparameters are being tested:
grep "learning_rate\|batch_size" /tmp/tuning_run.log | head -20
```

---

## Model-Specific Notes

### DQN
- **Trials**: 50
- **Duration**: 2.5-3 hours
- **Memory**: ~150MB VRAM (safe for RTX 3050 Ti)
- **Status**: Running (Trial 12/50)

### PPO
- **Trials**: 50
- **Duration**: 3-4 hours
- **Memory**: ~200MB VRAM
- **Recommendation**: Launch after DQN completes (19:25 CEST)

### TFT
- **Trials**: 50
- **Duration**: 4-5 hours
- **Memory**: ~1.8GB VRAM (45% of 4GB)
- **Warning**: Monitor for OOM, reduce batch size to 16 if needed

### MAMBA-2
- **Trials**: 50
- **Duration**: 2-3 hours
- **Memory**: ~2.5GB VRAM (62% of 4GB)
- **Warning**: Gradient checkpointing enabled, may still OOM with batch_size=16

### Liquid
- **Trials**: 50
- **Duration**: 1.5-2 hours
- **Memory**: ~120MB VRAM (safe)
- **Note**: Fastest model to tune

---

## Timeline (Estimated)

| Model | Start | End | Duration |
|-------|-------|-----|----------|
| DQN | 16:57 | 19:25 | 2.5h |
| PPO | 19:25 | 22:45 | 3.2h |
| TFT | 22:45 | 02:55 | 4.2h |
| MAMBA-2 | 02:55 | 05:00 | 2.1h |
| Liquid | 05:00 | 06:40 | 1.7h |
| **TOTAL** | **16:57** | **06:40** | **13.7h** |

All times in CEST (Central European Summer Time)

---

## Production Training (After Tuning)

Once best hyperparameters are identified:

```bash
# 1. Update model configs with best hyperparameters
vim ml/src/trainers/dqn.rs  # Update DQNHyperparameters::default()
vim ml/src/trainers/ppo.rs  # Update PpoHyperparameters::default()
# ... etc for other models

# 2. Run production training (4-6 weeks)
cargo run -p ml --example retrain_all_models --release --features cuda

# 3. Validate models
cargo run -p ml --example comprehensive_model_backtest --release
```

---

## Support

**Documentation**:
- Full status: `/home/jgrusewski/Work/foxhunt/HYPERPARAMETER_TUNING_STATUS.md`
- This guide: `/home/jgrusewski/Work/foxhunt/TUNING_QUICKSTART_GUIDE.md`
- Tuning config: `/home/jgrusewski/Work/foxhunt/tuning_config.yaml`

**Scripts**:
- Monitor: `/home/jgrusewski/Work/foxhunt/scripts/monitor_tuning.sh`
- Auto PPO: `/home/jgrusewski/Work/foxhunt/scripts/auto_launch_ppo.sh`
- Sequential: `/home/jgrusewski/Work/foxhunt/scripts/sequential_tuning_launcher.sh`

**Logs**:
- DQN: `/tmp/tuning_run.log`
- PPO: `/tmp/ppo_tuning_run.log`
- TFT: `/tmp/tft_tuning_run.log`
- MAMBA-2: `/tmp/mamba2_tuning_run.log`
- Liquid: `/tmp/liquid_tuning_run.log`

---

**Last Updated**: 2025-10-14 17:33 CEST
**Agent**: Agent 79
**Status**: DQN 24% complete (Trial 12/50)