267 lines
6.8 KiB
Markdown
267 lines
6.8 KiB
Markdown
# upload_binary.py - Quick Binary Upload Tool
|
|
|
|
**Status**: ✅ PRODUCTION READY
|
|
**Purpose**: Upload Rust release binaries to RunPod S3 for GPU training
|
|
**Location**: `scripts/upload_binary.py`
|
|
|
|
---
|
|
|
|
## Quick Start
|
|
|
|
```bash
|
|
# 1. Activate virtual environment
|
|
source .venv/bin/activate
|
|
|
|
# 2. Upload binary (auto-finds in target/release/examples/)
|
|
python3 scripts/upload_binary.py --binary-name hyperopt_mamba2_demo
|
|
|
|
# Output:
|
|
# 🔍 Step 1: Locating binary
|
|
# ✓ Found: /home/jgrusewski/Work/foxhunt/target/release/examples/hyperopt_mamba2_demo
|
|
#
|
|
# ✅ Step 2: Validating binary
|
|
# ✓ Valid executable
|
|
# Size: 14.2 MB (14,849,744 bytes)
|
|
#
|
|
# 📝 Step 3: Generating S3 key
|
|
# ✓ S3 key: binaries/hyperopt_mamba2_demo_cuda_20251030_120534
|
|
#
|
|
# 📤 Step 5: Uploading binary
|
|
#
|
|
# Upload Plan:
|
|
# Source: /home/.../hyperopt_mamba2_demo
|
|
# Destination: s3://se3zdnb5o4/binaries/hyperopt_mamba2_demo_cuda_20251030_120534
|
|
# Size: 14.2 MB (14,849,744 bytes)
|
|
# Force: False
|
|
#
|
|
# ↻ Upload required: File not found in S3
|
|
#
|
|
# Uploading hyperopt_mamba2_demo ━━━━━━━━━━ 100% • 14.2 MB • 45.3 MB/s • 0:00:00
|
|
#
|
|
# ✅ Upload complete!
|
|
# S3 URI: s3://se3zdnb5o4/binaries/hyperopt_mamba2_demo_cuda_20251030_120534
|
|
#
|
|
# Next steps:
|
|
# 1. Binary available at: /runpod-volume/binaries/hyperopt_mamba2_demo_cuda_20251030_120534
|
|
# 2. Use in deployment: --command '/runpod-volume/binaries/hyperopt_mamba2_demo_cuda_20251030_120534 --epochs 50'
|
|
# 3. Verify: aws s3 ls s3://se3zdnb5o4/binaries/hyperopt_mamba2_demo_cuda_20251030_120534 --profile runpod
|
|
```
|
|
|
|
---
|
|
|
|
## Common Commands
|
|
|
|
```bash
|
|
# Upload with auto-find (default)
|
|
python3 scripts/upload_binary.py --binary-name hyperopt_mamba2_demo
|
|
|
|
# Force overwrite (skip MD5 check)
|
|
python3 scripts/upload_binary.py --binary-name hyperopt_tft_demo --force
|
|
|
|
# Static name (no timestamp, overwrites)
|
|
python3 scripts/upload_binary.py --binary-name hyperopt_dqn_demo --no-timestamp
|
|
|
|
# Dry run (validate only, no upload)
|
|
python3 scripts/upload_binary.py --binary-name hyperopt_ppo_demo --dry-run
|
|
|
|
# Direct path
|
|
python3 scripts/upload_binary.py --binary-path ./target/release/examples/custom_binary
|
|
```
|
|
|
|
---
|
|
|
|
## Features
|
|
|
|
### ✅ Automatic Binary Location
|
|
- Searches `target/release/examples/` by name
|
|
- Handles build hashes (e.g., `-84b145a77f64618b`)
|
|
- Selects most recent if multiple matches
|
|
|
|
### ✅ Smart Upload
|
|
- **MD5 Checksum**: Skips upload if file unchanged
|
|
- **Progress Bar**: Shows transfer speed and ETA
|
|
- **Force Mode**: `--force` to always upload
|
|
|
|
### ✅ Flexible Naming
|
|
- **Timestamped** (default): `hyperopt_mamba2_demo_cuda_20251030_120534`
|
|
- **Static** (`--no-timestamp`): `hyperopt_mamba2_demo_cuda`
|
|
- **Plain** (`--no-timestamp --no-cuda`): `hyperopt_mamba2_demo`
|
|
|
|
### ✅ Validation
|
|
- Checks file exists and is executable
|
|
- Validates size (warns if < 100KB)
|
|
- Returns S3 path for deployment
|
|
|
|
---
|
|
|
|
## Options
|
|
|
|
| Flag | Description | Example |
|
|
|------|-------------|---------|
|
|
| `--binary-name NAME` | Auto-find binary | `hyperopt_mamba2_demo` |
|
|
| `--binary-path PATH` | Direct path | `./custom_binary` |
|
|
| `--force` | Skip checksum, force upload | `--force` |
|
|
| `--no-timestamp` | Static name (overwrites) | `--no-timestamp` |
|
|
| `--no-cuda` | Omit `_cuda` suffix | `--no-cuda` |
|
|
| `--dry-run` | Validate only, no upload | `--dry-run` |
|
|
|
|
---
|
|
|
|
## Requirements
|
|
|
|
### 1. Virtual Environment
|
|
```bash
|
|
source .venv/bin/activate
|
|
```
|
|
|
|
### 2. Dependencies
|
|
```bash
|
|
pip install -r foxhunt_runpod/foxhunt_runpod/requirements.txt
|
|
```
|
|
|
|
Includes:
|
|
- `boto3` - S3 uploads
|
|
- `rich` - Progress bars
|
|
- `pydantic-settings` - Config validation
|
|
- `python-dotenv` - .env loading
|
|
|
|
### 3. Configuration (.env.runpod)
|
|
```bash
|
|
RUNPOD_S3_ACCESS_KEY=<access_key>
|
|
RUNPOD_S3_SECRET=<secret_key>
|
|
RUNPOD_VOLUME_ID=se3zdnb5o4
|
|
RUNPOD_S3_ENDPOINT=https://s3api-eur-is-1.runpod.io
|
|
RUNPOD_S3_REGION=eur-is-1
|
|
```
|
|
|
|
---
|
|
|
|
## Workflow: Build → Upload → Deploy
|
|
|
|
```bash
|
|
# Step 1: Build binary
|
|
cargo build --release --example hyperopt_mamba2_demo --features cuda
|
|
|
|
# Step 2: Upload to S3
|
|
python3 scripts/upload_binary.py --binary-name hyperopt_mamba2_demo
|
|
# Returns: /runpod-volume/binaries/hyperopt_mamba2_demo_cuda_20251030_120534
|
|
|
|
# Step 3: Deploy to RunPod
|
|
python3 scripts/runpod_deploy.py \
|
|
--gpu-type "RTX A4000" \
|
|
--command "/runpod-volume/binaries/hyperopt_mamba2_demo_cuda_20251030_120534 \
|
|
--parquet-file /runpod-volume/test_data/ES_FUT_180d.parquet \
|
|
--trials 50 \
|
|
--timeout 2h"
|
|
```
|
|
|
|
---
|
|
|
|
## S3 Organization
|
|
|
|
```
|
|
s3://se3zdnb5o4/binaries/
|
|
├── hyperopt_mamba2_demo_cuda_20251030_120000
|
|
├── hyperopt_mamba2_demo_cuda_20251030_143000 (versioned)
|
|
├── hyperopt_tft_demo_cuda_20251030_150000
|
|
├── hyperopt_dqn_demo_cuda_20251030_163000
|
|
└── hyperopt_ppo_demo_cuda_20251030_173000
|
|
```
|
|
|
|
### Runpod Volume Mount
|
|
All binaries available at: `/runpod-volume/binaries/`
|
|
|
|
---
|
|
|
|
## Performance
|
|
|
|
| Binary | Size | Upload Time (50 Mbps) | MD5 Check |
|
|
|--------|------|----------------------|-----------|
|
|
| hyperopt_mamba2_demo | 14.2 MB | ~2.3s | 0s (skip) |
|
|
| hyperopt_tft_demo | 21.1 MB | ~3.4s | 0s (skip) |
|
|
| hyperopt_dqn_demo | 13.3 MB | ~2.1s | 0s (skip) |
|
|
| hyperopt_ppo_demo | 13.0 MB | ~2.1s | 0s (skip) |
|
|
|
|
**MD5 Optimization**: If binary unchanged, upload skipped (saves bandwidth)
|
|
|
|
---
|
|
|
|
## Troubleshooting
|
|
|
|
### "Binary not found"
|
|
```bash
|
|
# Build first
|
|
cargo build --release --example hyperopt_mamba2_demo --features cuda
|
|
|
|
# Verify
|
|
ls -lh target/release/examples/hyperopt_mamba2_demo
|
|
```
|
|
|
|
### "Not running in virtual environment"
|
|
```bash
|
|
source .venv/bin/activate
|
|
```
|
|
|
|
### "Configuration error"
|
|
```bash
|
|
# Check .env.runpod
|
|
cat .env.runpod | grep RUNPOD_S3
|
|
|
|
# Test S3 access
|
|
aws s3 ls s3://se3zdnb5o4/binaries/ \
|
|
--profile runpod \
|
|
--endpoint-url https://s3api-eur-is-1.runpod.io
|
|
```
|
|
|
|
---
|
|
|
|
## Advanced Usage
|
|
|
|
### Upload All Hyperopt Binaries
|
|
```bash
|
|
for binary in hyperopt_mamba2_demo hyperopt_tft_demo hyperopt_dqn_demo hyperopt_ppo_demo; do
|
|
python3 scripts/upload_binary.py --binary-name $binary
|
|
done
|
|
```
|
|
|
|
### Static Name for Production
|
|
```bash
|
|
# Always uploads to same path (simplifies deployment)
|
|
python3 scripts/upload_binary.py \
|
|
--binary-name hyperopt_mamba2_demo \
|
|
--no-timestamp \
|
|
--force
|
|
|
|
# Deploy script can use fixed path
|
|
--command "/runpod-volume/binaries/hyperopt_mamba2_demo_cuda"
|
|
```
|
|
|
|
---
|
|
|
|
## Documentation
|
|
|
|
- **BINARY_UPLOAD_QUICK_REF.md** - Quick reference guide
|
|
- **UPLOAD_BINARY_IMPLEMENTATION.md** - Full implementation details
|
|
- **RUNPOD_VOLUME_MOUNT_ARCHITECTURE.md** - S3 volume system
|
|
- **RUNPOD_DEPLOY_SCRIPT_UPDATE.md** - Deployment workflow
|
|
|
|
---
|
|
|
|
## Related Scripts
|
|
|
|
- **runpod_deploy.py** - Deploy pods with uploaded binaries
|
|
- **monitor_hyperopt.sh** - Monitor training progress
|
|
- **check_hyperopt_status.sh** - Check S3 results
|
|
|
|
---
|
|
|
|
## Version
|
|
|
|
**v1.0.0** (2025-10-30)
|
|
- ✅ Auto-locate binaries
|
|
- ✅ MD5 checksum validation
|
|
- ✅ Progress tracking
|
|
- ✅ Timestamped naming
|
|
- ✅ Dry run mode
|
|
- ✅ foxhunt_runpod integration
|