10f9cfadb7
Add train_launcher.sh that detects local GPU VRAM and routes training to local or Scaleway cloud. Auto-selects batch size per model based on available VRAM tier. Maps model names to actual ml/examples/train_*.rs cargo targets. Document Scaleway GPU instance types, setup procedure, batch size tables, and cost estimates. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
154 lines
4.5 KiB
Markdown
154 lines
4.5 KiB
Markdown
# Scaleway GPU Training Infrastructure
|
|
|
|
## Instance Types
|
|
|
|
| Instance | GPU | VRAM | vCPUs | RAM | Cost/hr |
|
|
|----------|-----|------|-------|-----|---------|
|
|
| GPU-3070-S | RTX 3070 | 8GB | 8 | 32GB | ~EUR 0.90 |
|
|
| L4-1-24G | L4 | 24GB | 8 | 48GB | ~EUR 1.20 |
|
|
| L4-2-48G | 2x L4 | 48GB | 16 | 96GB | ~EUR 2.40 |
|
|
|
|
**Recommendation:** GPU-3070-S for single-model training runs. L4-1-24G for hyperopt parallel trials and large batch training.
|
|
|
|
## Setup
|
|
|
|
### 1. Instance provisioning
|
|
|
|
```bash
|
|
# Create GPU instance via Scaleway CLI
|
|
scw instance server create \
|
|
type=GPU-3070-S \
|
|
image=ubuntu_jammy \
|
|
name=foxhunt-training \
|
|
root-volume=l:100G
|
|
```
|
|
|
|
### 2. Initial setup (run once after provisioning)
|
|
|
|
```bash
|
|
# Install Rust toolchain
|
|
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
|
|
source ~/.cargo/env
|
|
|
|
# Install CUDA toolkit (Ubuntu 22.04)
|
|
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb
|
|
sudo dpkg -i cuda-keyring_1.1-1_all.deb
|
|
sudo apt-get update
|
|
sudo apt-get -y install cuda-toolkit-12-4
|
|
|
|
# Clone and build
|
|
git clone ssh://gitea@git.fxhnt.ai:2222/foxhunt/foxhunt.git /opt/foxhunt
|
|
cd /opt/foxhunt
|
|
SQLX_OFFLINE=true cargo build --release -p ml --features cuda
|
|
```
|
|
|
|
### 3. Tailscale access
|
|
|
|
```bash
|
|
# Install Tailscale for secure access
|
|
curl -fsSL https://tailscale.com/install.sh | sh
|
|
sudo tailscale up --ssh
|
|
# Add to Tailscale ACL: tag:gpu-training
|
|
```
|
|
|
|
### 4. DNS (optional)
|
|
|
|
Add a Tailscale MagicDNS entry or Scaleway DNS record:
|
|
|
|
```
|
|
gpu.fxhnt.ai -> <tailscale-ip>
|
|
```
|
|
|
|
The `train_launcher.sh` script defaults to `FOXHUNT_GPU_HOST=gpu.fxhnt.ai`.
|
|
|
|
## Training workflow
|
|
|
|
### Local development then cloud production
|
|
|
|
```bash
|
|
# 1. Develop and test locally (RTX 3050 Ti, 4GB)
|
|
./scripts/train_launcher.sh --model dqn --epochs 5
|
|
|
|
# 2. Push code to Gitea
|
|
git push origin feat/model-improvements
|
|
|
|
# 3. Train on cloud GPU
|
|
./scripts/train_launcher.sh --model dqn --epochs 100 --cloud
|
|
|
|
# 4. Sync trained model back
|
|
rsync -avz training@gpu.fxhnt.ai:/opt/foxhunt/checkpoints/ ./checkpoints/
|
|
rsync -avz training@gpu.fxhnt.ai:/opt/foxhunt/ml/trained_models/ ./ml/trained_models/
|
|
```
|
|
|
|
### Model artifact sync
|
|
|
|
```bash
|
|
# Upload training data to cloud
|
|
rsync -avz ./data/databento/ training@gpu.fxhnt.ai:/opt/foxhunt/data/databento/
|
|
rsync -avz ./test_data/ training@gpu.fxhnt.ai:/opt/foxhunt/test_data/
|
|
|
|
# Download trained models
|
|
rsync -avz training@gpu.fxhnt.ai:/opt/foxhunt/checkpoints/ ./checkpoints/
|
|
rsync -avz training@gpu.fxhnt.ai:/opt/foxhunt/ml/trained_models/ ./ml/trained_models/
|
|
```
|
|
|
|
## Batch sizes by model and GPU
|
|
|
|
These values are used by `scripts/train_launcher.sh` auto-detection.
|
|
See `scripts/measure_vram.sh` for VRAM profiling methodology.
|
|
|
|
| Model | RTX 3050 Ti (4GB) | RTX 3070 (8GB) | L4 (24GB) |
|
|
|-------|-------------------|----------------|-----------|
|
|
| DQN | 128 | 256 | 512 |
|
|
| PPO | 230 (max) | 512 | 1024 |
|
|
| TFT | 32 | 64 | 256 |
|
|
| Mamba2 | 64 | 128 | 512 |
|
|
| CfC | 128 | 256 | 512 |
|
|
|
|
Notes:
|
|
- PPO on 4GB is capped at 230 batches (empirically verified, see MEMORY.md).
|
|
- TFT and Mamba2 are memory-intensive due to attention/state-space layers.
|
|
- All values assume `--features cuda` and `--release` builds.
|
|
|
|
## Cargo example targets
|
|
|
|
The launcher maps model names to these `ml` crate examples:
|
|
|
|
| Model | Cargo example | Source |
|
|
|-------|--------------|--------|
|
|
| dqn | `train_dqn` | `ml/examples/train_dqn.rs` |
|
|
| ppo | `train_ppo_parquet` | `ml/examples/train_ppo_parquet.rs` |
|
|
| tft | `train_tft_parquet` | `ml/examples/train_tft_parquet.rs` |
|
|
| mamba2 | `train_mamba2_parquet` | `ml/examples/train_mamba2_parquet.rs` |
|
|
| cfc | `train_liquid_dbn` | `ml/examples/train_liquid_dbn.rs` |
|
|
|
|
## Cost estimates
|
|
|
|
| Training run | Instance | Duration | Est. cost |
|
|
|-------------|----------|----------|-----------|
|
|
| DQN 100 epochs | GPU-3070-S | ~2 hours | ~EUR 1.80 |
|
|
| PPO 100 epochs | GPU-3070-S | ~3 hours | ~EUR 2.70 |
|
|
| TFT 100 epochs | L4-1-24G | ~4 hours | ~EUR 4.80 |
|
|
| Full hyperopt (50 trials) | L4-1-24G | ~12 hours | ~EUR 14.40 |
|
|
| All 5 models training | L4-1-24G | ~8 hours | ~EUR 9.60 |
|
|
|
|
## Environment variables
|
|
|
|
| Variable | Default | Description |
|
|
|----------|---------|-------------|
|
|
| `FOXHUNT_GPU_HOST` | `gpu.fxhnt.ai` | Cloud GPU hostname |
|
|
| `FOXHUNT_CLOUD_DIR` | `/opt/foxhunt` | Remote project directory |
|
|
| `SQLX_OFFLINE` | `true` | Required (no PostgreSQL on GPU instances) |
|
|
|
|
## Shutdown procedure
|
|
|
|
GPU instances are billed per hour. Always stop when not training:
|
|
|
|
```bash
|
|
# From local machine
|
|
scw instance server stop foxhunt-training
|
|
|
|
# Or from the instance itself
|
|
sudo shutdown -h now
|
|
```
|