# Scaleway GPU Training Infrastructure

## Instance Types

| Instance | GPU | VRAM | vCPUs | RAM | Cost/hr |
|----------|-----|------|-------|-----|---------|
| GPU-3070-S | RTX 3070 | 8GB | 8 | 32GB | ~EUR 0.90 |
| L4-1-24G | L4 | 24GB | 8 | 48GB | ~EUR 1.20 |
| L4-2-48G | 2x L4 | 48GB | 16 | 96GB | ~EUR 2.40 |

**Recommendation:** GPU-3070-S for single-model training runs. L4-1-24G for hyperopt parallel trials and large batch training.

## Setup

### 1. Instance provisioning

```bash
# Create GPU instance via Scaleway CLI
scw instance server create \
  type=GPU-3070-S \
  image=ubuntu_jammy \
  name=foxhunt-training \
  root-volume=l:100G
```

### 2. Initial setup (run once after provisioning)

```bash
# Install Rust toolchain
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
source ~/.cargo/env

# Install CUDA toolkit (Ubuntu 22.04)
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt-get update
sudo apt-get -y install cuda-toolkit-12-4

# Clone and build
git clone ssh://gitea@git.fxhnt.ai:2222/foxhunt/foxhunt.git /opt/foxhunt
cd /opt/foxhunt
SQLX_OFFLINE=true cargo build --release -p ml --features cuda
```

### 3. Tailscale access

```bash
# Install Tailscale for secure access
curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up --ssh
# Add to Tailscale ACL: tag:gpu-training
```

### 4. DNS (optional)

Add a Tailscale MagicDNS entry or Scaleway DNS record:

```
gpu.fxhnt.ai  ->  <tailscale-ip>
```

The `train_launcher.sh` script defaults to `FOXHUNT_GPU_HOST=gpu.fxhnt.ai`.

## Training workflow

### Local development then cloud production

```bash
# 1. Develop and test locally (RTX 3050 Ti, 4GB)
./scripts/train_launcher.sh --model dqn --epochs 5

# 2. Push code to Gitea
git push origin feat/model-improvements

# 3. Train on cloud GPU
./scripts/train_launcher.sh --model dqn --epochs 100 --cloud

# 4. Sync trained model back
rsync -avz training@gpu.fxhnt.ai:/opt/foxhunt/checkpoints/ ./checkpoints/
rsync -avz training@gpu.fxhnt.ai:/opt/foxhunt/ml/trained_models/ ./ml/trained_models/
```

### Model artifact sync

```bash
# Upload training data to cloud
rsync -avz ./data/databento/ training@gpu.fxhnt.ai:/opt/foxhunt/data/databento/
rsync -avz ./test_data/ training@gpu.fxhnt.ai:/opt/foxhunt/test_data/

# Download trained models
rsync -avz training@gpu.fxhnt.ai:/opt/foxhunt/checkpoints/ ./checkpoints/
rsync -avz training@gpu.fxhnt.ai:/opt/foxhunt/ml/trained_models/ ./ml/trained_models/
```

## Batch sizes by model and GPU

These values are used by `scripts/train_launcher.sh` auto-detection.
See `scripts/measure_vram.sh` for VRAM profiling methodology.

| Model | RTX 3050 Ti (4GB) | RTX 3070 (8GB) | L4 (24GB) |
|-------|-------------------|----------------|-----------|
| DQN | 128 | 256 | 512 |
| PPO | 230 (max) | 512 | 1024 |
| TFT | 32 | 64 | 256 |
| Mamba2 | 64 | 128 | 512 |
| CfC | 128 | 256 | 512 |

Notes:
- PPO on 4GB is capped at 230 batches (empirically verified, see MEMORY.md).
- TFT and Mamba2 are memory-intensive due to attention/state-space layers.
- All values assume `--features cuda` and `--release` builds.

## Cargo example targets

The launcher maps model names to these `ml` crate examples:

| Model | Cargo example | Source |
|-------|--------------|--------|
| dqn | `train_dqn` | `ml/examples/train_dqn.rs` |
| ppo | `train_ppo_parquet` | `ml/examples/train_ppo_parquet.rs` |
| tft | `train_tft_parquet` | `ml/examples/train_tft_parquet.rs` |
| mamba2 | `train_mamba2_parquet` | `ml/examples/train_mamba2_parquet.rs` |
| cfc | `train_liquid_dbn` | `ml/examples/train_liquid_dbn.rs` |

## Cost estimates

| Training run | Instance | Duration | Est. cost |
|-------------|----------|----------|-----------|
| DQN 100 epochs | GPU-3070-S | ~2 hours | ~EUR 1.80 |
| PPO 100 epochs | GPU-3070-S | ~3 hours | ~EUR 2.70 |
| TFT 100 epochs | L4-1-24G | ~4 hours | ~EUR 4.80 |
| Full hyperopt (50 trials) | L4-1-24G | ~12 hours | ~EUR 14.40 |
| All 5 models training | L4-1-24G | ~8 hours | ~EUR 9.60 |

## Environment variables

| Variable | Default | Description |
|----------|---------|-------------|
| `FOXHUNT_GPU_HOST` | `gpu.fxhnt.ai` | Cloud GPU hostname |
| `FOXHUNT_CLOUD_DIR` | `/opt/foxhunt` | Remote project directory |
| `SQLX_OFFLINE` | `true` | Required (no PostgreSQL on GPU instances) |

## Shutdown procedure

GPU instances are billed per hour. Always stop when not training:

```bash
# From local machine
scw instance server stop foxhunt-training

# Or from the instance itself
sudo shutdown -h now
```