foxhunt/BINARY_UPLOAD_QUICK_REF.md

# Binary Upload Quick Reference

**Script**: `scripts/upload_binary.py`
**Purpose**: Quick binary uploads to RunPod S3 for hyperparameter optimization workflows
**Status**: ✅ PRODUCTION READY
**Last Updated**: 2025-10-30

---

## Quick Start

```bash
# 1. Activate .venv
source .venv/bin/activate

# 2. Upload binary (auto-finds in target/release/examples/)
python3 scripts/upload_binary.py --binary-name hyperopt_mamba2_demo

# Output:
# ✅ Uploaded to: s3://se3zdnb5o4/binaries/hyperopt_mamba2_demo_cuda_20251030_001234
# Next: /runpod-volume/binaries/hyperopt_mamba2_demo_cuda_20251030_001234
```

---

## Common Use Cases

### 1. Upload Latest Hyperopt Binary (Default)
```bash
python3 scripts/upload_binary.py --binary-name hyperopt_mamba2_demo
# → binaries/hyperopt_mamba2_demo_cuda_20251030_120534
```

### 2. Force Overwrite (Skip Checksum)
```bash
python3 scripts/upload_binary.py --binary-name hyperopt_tft_demo --force
# Uploads even if MD5 matches
```

### 3. Upload Without Timestamp (Static Name)
```bash
python3 scripts/upload_binary.py --binary-name hyperopt_dqn_demo --no-timestamp
# → binaries/hyperopt_dqn_demo_cuda
# ⚠️ Overwrites existing file!
```

### 4. Upload Non-CUDA Binary
```bash
python3 scripts/upload_binary.py --binary-name custom_tool --no-cuda
# → binaries/custom_tool_20251030_120534
```

### 5. Dry Run (Validation Only)
```bash
python3 scripts/upload_binary.py --binary-name hyperopt_ppo_demo --dry-run
# Validates binary but doesn't upload
```

### 6. Upload Custom Path
```bash
python3 scripts/upload_binary.py --binary-path ./my_custom_binary --force
# Upload from anywhere
```

---

## Features

### Automatic Binary Location
- Searches `target/release/examples/` by name
- Handles build hashes (e.g., `hyperopt_mamba2_demo-84b145a77f64618b`)
- Selects most recent if multiple matches

### Validation
- ✅ Checks file exists and is executable
- ✅ Validates size (warns if < 100KB)
- ✅ MD5 checksum comparison (skips upload if unchanged)

### S3 Organization
```
s3://se3zdnb5o4/binaries/
├── hyperopt_mamba2_demo_cuda_20251030_120000
├── hyperopt_tft_demo_cuda_20251030_143000
├── hyperopt_dqn_demo_cuda_20251030_150000
└── hyperopt_ppo_demo_cuda_20251030_163000
```

### Progress Tracking
```
Uploading hyperopt_mamba2_demo ━━━━━━━━━━ 100% • 14.2 MB • 45.3 MB/s • 0:00:00
✅ Upload complete!
  S3 URI: s3://se3zdnb5o4/binaries/hyperopt_mamba2_demo_cuda_20251030_120534
```

---

## Integration with Deployment

### Step 1: Build Binary
```bash
cargo build --release --example hyperopt_mamba2_demo --features cuda
```

### Step 2: Upload to S3
```bash
python3 scripts/upload_binary.py --binary-name hyperopt_mamba2_demo
# → /runpod-volume/binaries/hyperopt_mamba2_demo_cuda_20251030_120534
```

### Step 3: Deploy to RunPod
```bash
python3 scripts/runpod_deploy.py \
  --gpu-type "RTX A4000" \
  --command "/runpod-volume/binaries/hyperopt_mamba2_demo_cuda_20251030_120534 \
    --parquet-file /runpod-volume/test_data/ES_FUT_180d.parquet \
    --trials 50 \
    --timeout 2h \
    --s3-bucket se3zdnb5o4 \
    --s3-prefix hyperopt_runs/mamba2/"
```

---

## Options Reference

| Option | Description | Example |
|---|---|---|
| `--binary-name` | Binary name (auto-finds) | `hyperopt_mamba2_demo` |
| `--binary-path` | Direct path to binary | `./custom_binary` |
| `--force` | Force upload (skip checksum) | `--force` |
| `--no-timestamp` | Static name (overwrites) | `--no-timestamp` |
| `--no-cuda` | Omit `_cuda` suffix | `--no-cuda` |
| `--dry-run` | Validate only | `--dry-run` |

---

## Requirements

### Environment
```bash
# Activate .venv (REQUIRED)
source .venv/bin/activate
```

### Configuration (.env.runpod)
```bash
RUNPOD_S3_ACCESS_KEY=<access_key>
RUNPOD_S3_SECRET=<secret_key>
RUNPOD_VOLUME_ID=se3zdnb5o4
RUNPOD_S3_ENDPOINT=https://s3api-eur-is-1.runpod.io
RUNPOD_S3_REGION=eur-is-1
```

### Dependencies
```bash
pip install -r foxhunt_runpod/foxhunt_runpod/requirements.txt
# Installs: boto3, rich, pydantic-settings, python-dotenv
```

---

## Troubleshooting

### "Binary not found"
```bash
# Build binary first
cargo build --release --example hyperopt_mamba2_demo --features cuda

# Verify location
ls -lh target/release/examples/hyperopt_mamba2_demo
```

### "Not running in virtual environment"
```bash
source .venv/bin/activate
python3 scripts/upload_binary.py --help
```

### "Configuration error"
```bash
# Verify .env.runpod exists
cat .env.runpod | grep RUNPOD_S3

# Check S3 credentials
aws s3 ls s3://se3zdnb5o4/binaries/ \
  --profile runpod \
  --endpoint-url https://s3api-eur-is-1.runpod.io
```

### "Binary suspiciously small"
```bash
# Check binary was built with --release
cargo build --release --example <name> --features cuda

# Debug build produces smaller, unoptimized binaries
```

---

## Workflow Examples

### Example 1: MAMBA-2 Hyperopt Iteration
```bash
# 1. Update code
vim ml/examples/hyperopt_mamba2_demo.rs

# 2. Rebuild
cargo build --release --example hyperopt_mamba2_demo --features cuda

# 3. Upload
python3 scripts/upload_binary.py --binary-name hyperopt_mamba2_demo
# → binaries/hyperopt_mamba2_demo_cuda_20251030_153400

# 4. Deploy
python3 scripts/runpod_deploy.py \
  --command "/runpod-volume/binaries/hyperopt_mamba2_demo_cuda_20251030_153400 \
    --trials 50 --timeout 2h"
```

### Example 2: Quick Overwrite (Same Binary Name)
```bash
# Fast iteration: overwrite with static name
cargo build --release --example hyperopt_dqn_demo --features cuda
python3 scripts/upload_binary.py \
  --binary-name hyperopt_dqn_demo \
  --no-timestamp \
  --force

# Deploy always uses same path
python3 scripts/runpod_deploy.py \
  --command "/runpod-volume/binaries/hyperopt_dqn_demo_cuda"
```

### Example 3: Multi-Binary Upload
```bash
# Upload all hyperopt binaries at once
for binary in hyperopt_mamba2_demo hyperopt_tft_demo hyperopt_dqn_demo hyperopt_ppo_demo; do
  python3 scripts/upload_binary.py --binary-name $binary
done
```

---

## Performance Notes

| Binary | Size | Upload Time (50 Mbps) | Typical Use |
|---|---|---|---|
| hyperopt_mamba2_demo | 14.2 MB | ~2.3s | MAMBA-2 hyperopt |
| hyperopt_tft_demo | 21.1 MB | ~3.4s | TFT hyperopt |
| hyperopt_dqn_demo | 13.3 MB | ~2.1s | DQN hyperopt |
| hyperopt_ppo_demo | 13.0 MB | ~2.1s | PPO hyperopt |

**MD5 Checksum**: If file unchanged, upload skipped (0s)

---

## Best Practices

1. **Always Use .venv**: Ensures correct dependencies
2. **Use Timestamps**: Allows version history (default)
3. **Force Only When Needed**: Saves bandwidth
4. **Dry Run First**: Validate before uploading large files
5. **Static Names for Stable Workflows**: Use `--no-timestamp` for production deployments

---

## Related Documentation

- **RUNPOD_VOLUME_MOUNT_ARCHITECTURE.md**: Volume mount system design
- **RUNPOD_DEPLOY_SCRIPT_UPDATE.md**: Full deployment workflow
- **ML_TRAINING_PARQUET_GUIDE.md**: Training binary usage
- **HYPEROPT_DEPLOYMENT_COMPLETE.md**: Hyperopt system architecture

---

## Version History

**v1.0.0** (2025-10-30)
- ✅ Auto-locates binaries in target/release/examples/
- ✅ MD5 checksum validation (skips unchanged)
- ✅ Timestamped naming for version control
- ✅ Progress bar with transfer speed
- ✅ Dry run mode for validation
- ✅ Integration with foxhunt_runpod.S3Client