foxhunt/docs/infra/scaleway-gpu-training.md

Scaleway GPU Training Infrastructure

Instance Types

Instance	GPU	VRAM	vCPUs	RAM	Cost/hr
GPU-3070-S	RTX 3070	8GB	8	32GB	~EUR 0.90
L4-1-24G	L4	24GB	8	48GB	~EUR 1.20
L4-2-48G	2x L4	48GB	16	96GB	~EUR 2.40

Recommendation: GPU-3070-S for single-model training runs. L4-1-24G for hyperopt parallel trials and large batch training.

Setup

1. Instance provisioning

# Create GPU instance via Scaleway CLI
scw instance server create \
  type=GPU-3070-S \
  image=ubuntu_jammy \
  name=foxhunt-training \
  root-volume=l:100G

2. Initial setup (run once after provisioning)

# Install Rust toolchain
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
source ~/.cargo/env

# Install CUDA toolkit (Ubuntu 22.04)
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt-get update
sudo apt-get -y install cuda-toolkit-12-4

# Clone and build
git clone ssh://gitea@git.fxhnt.ai:2222/foxhunt/foxhunt.git /opt/foxhunt
cd /opt/foxhunt
SQLX_OFFLINE=true cargo build --release -p ml --features cuda

3. Tailscale access

# Install Tailscale for secure access
curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up --ssh
# Add to Tailscale ACL: tag:gpu-training

4. DNS (optional)

Add a Tailscale MagicDNS entry or Scaleway DNS record:

gpu.fxhnt.ai  ->  <tailscale-ip>

The train_launcher.sh script defaults to FOXHUNT_GPU_HOST=gpu.fxhnt.ai.

Training workflow

Local development then cloud production

# 1. Develop and test locally (RTX 3050 Ti, 4GB)
./scripts/train_launcher.sh --model dqn --epochs 5

# 2. Push code to Gitea
git push origin feat/model-improvements

# 3. Train on cloud GPU
./scripts/train_launcher.sh --model dqn --epochs 100 --cloud

# 4. Sync trained model back
rsync -avz training@gpu.fxhnt.ai:/opt/foxhunt/checkpoints/ ./checkpoints/
rsync -avz training@gpu.fxhnt.ai:/opt/foxhunt/ml/trained_models/ ./ml/trained_models/

Model artifact sync

# Upload training data to cloud
rsync -avz ./data/databento/ training@gpu.fxhnt.ai:/opt/foxhunt/data/databento/
rsync -avz ./test_data/ training@gpu.fxhnt.ai:/opt/foxhunt/test_data/

# Download trained models
rsync -avz training@gpu.fxhnt.ai:/opt/foxhunt/checkpoints/ ./checkpoints/
rsync -avz training@gpu.fxhnt.ai:/opt/foxhunt/ml/trained_models/ ./ml/trained_models/

Batch sizes by model and GPU

These values are used by scripts/train_launcher.sh auto-detection. See scripts/measure_vram.sh for VRAM profiling methodology.

Model	RTX 3050 Ti (4GB)	RTX 3070 (8GB)	L4 (24GB)
DQN	128	256	512
PPO	230 (max)	512	1024
TFT	32	64	256
Mamba2	64	128	512
CfC	128	256	512

Notes:

• PPO on 4GB is capped at 230 batches (empirically verified, see MEMORY.md).
• TFT and Mamba2 are memory-intensive due to attention/state-space layers.
• All values assume --features cuda and --release builds.

Cargo example targets

The launcher maps model names to these ml crate examples:

Model	Cargo example	Source
dqn	train_dqn	ml/examples/train_dqn.rs
ppo	train_ppo_parquet	ml/examples/train_ppo_parquet.rs
tft	train_tft_parquet	ml/examples/train_tft_parquet.rs
mamba2	train_mamba2_parquet	ml/examples/train_mamba2_parquet.rs
cfc	train_liquid_dbn	ml/examples/train_liquid_dbn.rs

Cost estimates

Training run	Instance	Duration	Est. cost
DQN 100 epochs	GPU-3070-S	~2 hours	~EUR 1.80
PPO 100 epochs	GPU-3070-S	~3 hours	~EUR 2.70
TFT 100 epochs	L4-1-24G	~4 hours	~EUR 4.80
Full hyperopt (50 trials)	L4-1-24G	~12 hours	~EUR 14.40
All 5 models training	L4-1-24G	~8 hours	~EUR 9.60

Environment variables

Variable	Default	Description
FOXHUNT_GPU_HOST	gpu.fxhnt.ai	Cloud GPU hostname
FOXHUNT_CLOUD_DIR	/opt/foxhunt	Remote project directory
SQLX_OFFLINE	true	Required (no PostgreSQL on GPU instances)

Shutdown procedure

GPU instances are billed per hour. Always stop when not training:

# From local machine
scw instance server stop foxhunt-training

# Or from the instance itself
sudo shutdown -h now