Files

jgrusewski d73316da3d chore: Pre-cleanup commit - save current state before major reorganization

2025-10-30 00:54:01 +01:00

6.8 KiB

Raw Blame History

upload_binary.py - Quick Binary Upload Tool

Status: ✅ PRODUCTION READY Purpose: Upload Rust release binaries to RunPod S3 for GPU training Location: scripts/upload_binary.py

Quick Start

# 1. Activate virtual environment
source .venv/bin/activate

# 2. Upload binary (auto-finds in target/release/examples/)
python3 scripts/upload_binary.py --binary-name hyperopt_mamba2_demo

# Output:
# 🔍 Step 1: Locating binary
#   ✓ Found: /home/jgrusewski/Work/foxhunt/target/release/examples/hyperopt_mamba2_demo
#
# ✅ Step 2: Validating binary
#   ✓ Valid executable
#   Size: 14.2 MB (14,849,744 bytes)
#
# 📝 Step 3: Generating S3 key
#   ✓ S3 key: binaries/hyperopt_mamba2_demo_cuda_20251030_120534
#
# 📤 Step 5: Uploading binary
#
# Upload Plan:
#   Source:      /home/.../hyperopt_mamba2_demo
#   Destination: s3://se3zdnb5o4/binaries/hyperopt_mamba2_demo_cuda_20251030_120534
#   Size:        14.2 MB (14,849,744 bytes)
#   Force:       False
#
# ↻ Upload required: File not found in S3
#
# Uploading hyperopt_mamba2_demo ━━━━━━━━━━ 100% • 14.2 MB • 45.3 MB/s • 0:00:00
#
# ✅ Upload complete!
#   S3 URI: s3://se3zdnb5o4/binaries/hyperopt_mamba2_demo_cuda_20251030_120534
#
# Next steps:
#   1. Binary available at: /runpod-volume/binaries/hyperopt_mamba2_demo_cuda_20251030_120534
#   2. Use in deployment: --command '/runpod-volume/binaries/hyperopt_mamba2_demo_cuda_20251030_120534 --epochs 50'
#   3. Verify: aws s3 ls s3://se3zdnb5o4/binaries/hyperopt_mamba2_demo_cuda_20251030_120534 --profile runpod

Common Commands

# Upload with auto-find (default)
python3 scripts/upload_binary.py --binary-name hyperopt_mamba2_demo

# Force overwrite (skip MD5 check)
python3 scripts/upload_binary.py --binary-name hyperopt_tft_demo --force

# Static name (no timestamp, overwrites)
python3 scripts/upload_binary.py --binary-name hyperopt_dqn_demo --no-timestamp

# Dry run (validate only, no upload)
python3 scripts/upload_binary.py --binary-name hyperopt_ppo_demo --dry-run

# Direct path
python3 scripts/upload_binary.py --binary-path ./target/release/examples/custom_binary

Features

✅ Automatic Binary Location

Searches target/release/examples/ by name
Handles build hashes (e.g., -84b145a77f64618b)
Selects most recent if multiple matches

✅ Smart Upload

MD5 Checksum: Skips upload if file unchanged
Progress Bar: Shows transfer speed and ETA
Force Mode: --force to always upload

✅ Flexible Naming

Timestamped (default): hyperopt_mamba2_demo_cuda_20251030_120534
Static (--no-timestamp): hyperopt_mamba2_demo_cuda
Plain (--no-timestamp --no-cuda): hyperopt_mamba2_demo

✅ Validation

Checks file exists and is executable
Validates size (warns if < 100KB)
Returns S3 path for deployment

Options

Flag	Description	Example
`--binary-name NAME`	Auto-find binary	`hyperopt_mamba2_demo`
`--binary-path PATH`	Direct path	`./custom_binary`
`--force`	Skip checksum, force upload	`--force`
`--no-timestamp`	Static name (overwrites)	`--no-timestamp`
`--no-cuda`	Omit `_cuda` suffix	`--no-cuda`
`--dry-run`	Validate only, no upload	`--dry-run`

Requirements

1. Virtual Environment

source .venv/bin/activate

2. Dependencies

pip install -r foxhunt_runpod/foxhunt_runpod/requirements.txt

Includes:

boto3 - S3 uploads
rich - Progress bars
pydantic-settings - Config validation
python-dotenv - .env loading

3. Configuration (.env.runpod)

RUNPOD_S3_ACCESS_KEY=<access_key>
RUNPOD_S3_SECRET=<secret_key>
RUNPOD_VOLUME_ID=se3zdnb5o4
RUNPOD_S3_ENDPOINT=https://s3api-eur-is-1.runpod.io
RUNPOD_S3_REGION=eur-is-1

Workflow: Build → Upload → Deploy

# Step 1: Build binary
cargo build --release --example hyperopt_mamba2_demo --features cuda

# Step 2: Upload to S3
python3 scripts/upload_binary.py --binary-name hyperopt_mamba2_demo
# Returns: /runpod-volume/binaries/hyperopt_mamba2_demo_cuda_20251030_120534

# Step 3: Deploy to RunPod
python3 scripts/runpod_deploy.py \
  --gpu-type "RTX A4000" \
  --command "/runpod-volume/binaries/hyperopt_mamba2_demo_cuda_20251030_120534 \
    --parquet-file /runpod-volume/test_data/ES_FUT_180d.parquet \
    --trials 50 \
    --timeout 2h"

S3 Organization

s3://se3zdnb5o4/binaries/
├── hyperopt_mamba2_demo_cuda_20251030_120000
├── hyperopt_mamba2_demo_cuda_20251030_143000  (versioned)
├── hyperopt_tft_demo_cuda_20251030_150000
├── hyperopt_dqn_demo_cuda_20251030_163000
└── hyperopt_ppo_demo_cuda_20251030_173000

Runpod Volume Mount

All binaries available at: /runpod-volume/binaries/

Performance

Binary	Size	Upload Time (50 Mbps)	MD5 Check
hyperopt_mamba2_demo	14.2 MB	~2.3s	0s (skip)
hyperopt_tft_demo	21.1 MB	~3.4s	0s (skip)
hyperopt_dqn_demo	13.3 MB	~2.1s	0s (skip)
hyperopt_ppo_demo	13.0 MB	~2.1s	0s (skip)

MD5 Optimization: If binary unchanged, upload skipped (saves bandwidth)

Troubleshooting

"Binary not found"

# Build first
cargo build --release --example hyperopt_mamba2_demo --features cuda

# Verify
ls -lh target/release/examples/hyperopt_mamba2_demo

"Not running in virtual environment"

source .venv/bin/activate

"Configuration error"

# Check .env.runpod
cat .env.runpod | grep RUNPOD_S3

# Test S3 access
aws s3 ls s3://se3zdnb5o4/binaries/ \
  --profile runpod \
  --endpoint-url https://s3api-eur-is-1.runpod.io

Advanced Usage

Upload All Hyperopt Binaries

for binary in hyperopt_mamba2_demo hyperopt_tft_demo hyperopt_dqn_demo hyperopt_ppo_demo; do
  python3 scripts/upload_binary.py --binary-name $binary
done

Static Name for Production

# Always uploads to same path (simplifies deployment)
python3 scripts/upload_binary.py \
  --binary-name hyperopt_mamba2_demo \
  --no-timestamp \
  --force

# Deploy script can use fixed path
--command "/runpod-volume/binaries/hyperopt_mamba2_demo_cuda"

Documentation

BINARY_UPLOAD_QUICK_REF.md - Quick reference guide
UPLOAD_BINARY_IMPLEMENTATION.md - Full implementation details
RUNPOD_VOLUME_MOUNT_ARCHITECTURE.md - S3 volume system
RUNPOD_DEPLOY_SCRIPT_UPDATE.md - Deployment workflow

runpod_deploy.py - Deploy pods with uploaded binaries
monitor_hyperopt.sh - Monitor training progress
check_hyperopt_status.sh - Check S3 results

Version

v1.0.0 (2025-10-30)

✅ Auto-locate binaries
✅ MD5 checksum validation
✅ Progress tracking
✅ Timestamped naming
✅ Dry run mode
✅ foxhunt_runpod integration

6.8 KiB Raw Blame History