gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
Files
e61e8f54da130c2c6972e7f679add72e2b3a0959
foxhunt/DEPLOY_INDEX.md
jgrusewski aac0597cd2 feat(ml): DQN Option B checkpoint fix + TFT OOM investigation 
- Fixed DQN early stopping checkpoint naming bug (Option B)
  - Added is_final: bool parameter to checkpoint callback signature
  - Trainer now distinguishes final checkpoints from regular epoch checkpoints
  - Final checkpoints use 'dqn_final_epoch{N}' naming convention
  - Regular checkpoints use 'dqn_epoch_{N}' naming convention

- Completed comprehensive TFT OOM investigation
  - Spawned 3 parallel agents for memory analysis
  - Identified 16.4GB memory leak (29.7x over expected 525-550MB)
  - Root causes: Attention cache bloat (960MB), gradient accumulation bug, detached tensors
  - Recommended fixes: Disable cache during training, explicit tensor drops
  - Created TFT_MEMORY_ANALYSIS.md, TFT_MEMORY_LEAK_ANALYSIS.md

- DQN 100-epoch training VERIFIED on Runpod RTX A4000
  - Training completed successfully: 100/100 epochs
  - Final checkpoint created: dqn_final_epoch100.safetensors
  - Training speed: 4.8 sec/epoch (3.5x faster than baseline)
  - Option B fix working perfectly

- Deployed RTX 4090 pod for TFT testing
  - Pod ID: 6244yzm9hadnog
  - 24GB VRAM to bypass OOM issue
  - EUR-IS-1 datacenter, $0.59/hr

Files modified:
- ml/examples/train_dqn.rs (checkpoint callback signature)
- ml/src/trainers/dqn.rs (callback signature + is_final parameter)
- CLAUDE.md (compacted to ~11k chars)

Generated reports:
- TFT_MEMORY_ANALYSIS.md (15-section memory breakdown)
- TFT_MEMORY_QUICK_SUMMARY.md (executive summary)
- TFT_MEMORY_LEAK_ANALYSIS.md (5 critical leaks identified)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-25 23:49:24 +02:00

331 lines
10 KiB
Markdown
Raw Blame History

# Runpod Deployment Index
**Last Updated**: 2025-10-25T18:02:42Z
**Status**: ✅ **READY FOR DEPLOYMENT**
---
## Quick Navigation
### 1. Quick Start (Most Users)
- **DEPLOY_01_STATUS.txt** - Visual summary of deployment status (1 minute read)
- **RUNPOD_DEPLOYMENT_COMMANDS.md** - Copy-paste deployment commands (5 minute read)
### 2. Detailed Documentation
- **AGENT_DEPLOY_01_RUNPOD_UPLOAD.md** - Full deployment report (10 minute read)
- **AGENT_DEPLOY_01_QUICK_SUMMARY.md** - Quick reference guide (2 minute read)
### 3. Configuration Files
- **runpod_deployment_manifest.json** - Binary metadata with SHA-256 checksums
---
## File Descriptions
### DEPLOY_01_STATUS.txt (1.5 KB)
**Purpose**: Visual summary of deployment status with ASCII boxes
**Contents**:
- Binaries uploaded (5/5 with checksums)
- Verification results (6/6 success criteria)
- Production readiness metrics
- Quick start command for TFT training
- Next step: DEPLOY-02 (upload test data)
**When to use**: Quick status check, sharing with team
---
### RUNPOD_DEPLOYMENT_COMMANDS.md (15 KB)
**Purpose**: Copy-paste ready deployment commands for all scenarios
**Contents**:
1. **Prerequisites**: AWS CLI setup, Runpod account
2. **Option 1**: Single model training (TFT recommended)
- Pod configuration (RTX 4090, $0.44/hr)
- Complete startup script with error handling
- Expected output and cost (~$0.015 per run)
3. **Option 2**: All models sequential training
- Batch script for 4 models (~5 minutes total)
- Cost: ~$0.04 for all models
4. **Option 3**: Automated pod creation via Python API
- Python script for `runpod` CLI
- Auto-termination after training
5. **Verification Commands**: Post-training validation
6. **Cost Optimization Tips**: Spot instances (70% cheaper), batch training
7. **Troubleshooting**: Common errors and solutions
**When to use**: Creating Runpod pods, deploying to GPU
---
### AGENT_DEPLOY_01_RUNPOD_UPLOAD.md (18 KB)
**Purpose**: Complete deployment report with all technical details
**Contents**:
- **Phase 1**: Binary compilation results (5 binaries)
- **Phase 2**: Binary integrity verification (CUDA support, checksums)
- **Phase 3**: Runpod S3 upload (85.9 MB total, 7.0 MB/s)
- **Phase 4**: Deployment manifest creation
- **Deployment Commands**: 3 ready-to-use options
- **Success Criteria**: 6/6 validation results
- **Production Readiness**: Test coverage, GPU memory, binary specs
- **Recommended GPU Configurations**: Per-model requirements
- **Next Steps**: Immediate and short/medium-term actions
- **Appendix**: Troubleshooting guide
**When to use**: Understanding deployment process, debugging issues, technical review
---
### AGENT_DEPLOY_01_QUICK_SUMMARY.md (2 KB)
**Purpose**: Quick reference for key results and commands
**Contents**:
- What was done (5 bullet points)
- Key results table (5 binaries with checksums)
- Quick start commands (download, verify, deploy)
- Production readiness checklist
- Next steps (DEPLOY-02 onwards)
**When to use**: Quick refresher, sharing results with stakeholders
---
### runpod_deployment_manifest.json (1.4 KB)
**Purpose**: Machine-readable deployment metadata
**Contents**:
```json
{
"deployment_date": "2025-10-25T18:02:42Z",
"git_commit": "caf36b41...",
"binaries": [
{
"name": "train_dqn",
"size": 20857232,
"sha256": "fedc57ea...",
"s3_path": "s3://se3zdnb5o4/binaries/train_dqn"
},
// ... 4 more binaries
],
"test_pass_rate": "100% (1,337/1,337 ML, 3,196/3,196 total)",
"production_status": "CERTIFIED",
"cuda_support": true,
"models": ["DQN", "PPO", "MAMBA-2", "TFT-FP32"],
"features": 225
}
```
**When to use**: Automated scripts, CI/CD pipelines, verification tools
---
## Deployment Workflow
### Step 1: Compile Binaries (COMPLETE ✅)
- Status: All 5 binaries compiled
- Location: `target/release/examples/train_*`
- CUDA support: Verified (CUDA 12.9)
- See: **AGENT_DEPLOY_01_RUNPOD_UPLOAD.md** (Phase 1)
### Step 2: Upload to Runpod S3 (COMPLETE ✅)
- Status: All 5 binaries uploaded
- Location: `s3://se3zdnb5o4/binaries/`
- Total size: 85.9 MB
- Upload speed: 7.0 MB/s average
- See: **AGENT_DEPLOY_01_RUNPOD_UPLOAD.md** (Phase 3)
### Step 3: Upload Test Data (PENDING ⏳)
- Next agent: DEPLOY-02
- Files to upload:
- ES_FUT_180d.parquet (2.9 MB)
- NQ_FUT_180d.parquet (4.4 MB)
- 6E_FUT_180d.parquet (2.8 MB)
- ZN_FUT_90d.parquet (2.8 MB)
- Command:
```bash
aws s3 cp test_data/*.parquet s3://se3zdnb5o4/test_data/ \
--profile runpod \
--endpoint-url https://s3api-eur-is-1.runpod.io
```
### Step 4: Create Runpod Pod (PENDING ⏳)
- Next agent: DEPLOY-03
- Use: **RUNPOD_DEPLOYMENT_COMMANDS.md** (Option 1)
- GPU: NVIDIA RTX 4090 (24GB VRAM)
- Cost: $0.44/hr (~$0.015 per 2-minute TFT run)
### Step 5: Run Training (PENDING ⏳)
- Next agent: DEPLOY-04
- Expected time: ~2 minutes (TFT), ~5 minutes (all 4 models)
- Expected cost: ~$0.015 (TFT), ~$0.04 (all models)
### Step 6: Validate Checkpoints (PENDING ⏳)
- Next agent: DEPLOY-05
- Download checkpoints from `s3://se3zdnb5o4/models/`
- Run local inference tests
- Verify model accuracy (RMSE <0.05 for 225 features)
---
## Binary Details
| Binary | Size | Purpose | GPU Memory | Training Time |
|--------|------|---------|------------|---------------|
| train_dqn | 19.9 MB | Deep Q-Network | ~6 MB | 15-20 sec |
| train_ppo | 12.5 MB | Proximal Policy Opt. | ~145 MB | 7-10 sec |
| train_mamba2_dbn | 13.3 MB | MAMBA-2 (DBN data) | ~164 MB | 2-3 min |
| train_mamba2_parquet | 19.7 MB | MAMBA-2 (Parquet) | ~164 MB | 2-3 min |
| train_tft_parquet | 20.6 MB | TFT (Parquet, cache opt.) | ~525-550 MB | 2 min |
**Total GPU Memory (All Models)**: 840-865 MB (fits on RTX 4090/3060/A4000)
---
## Cost Estimates
### Single Model Training (TFT)
- GPU: NVIDIA RTX 4090 (24GB VRAM)
- Training time: ~2 minutes
- Cost per run: **$0.015** ($0.44/hr * 2/60 hr)
- Monthly cost (daily retraining): **$0.45** ($0.015 * 30 days)
### Batch Training (All 4 Models)
- GPU: NVIDIA RTX 4090 (24GB VRAM)
- Training time: ~5 minutes (DQN 20s + PPO 10s + MAMBA-2 3min + TFT 2min)
- Cost per run: **$0.04** ($0.44/hr * 5/60 hr)
- Monthly cost (daily retraining): **$1.20** ($0.04 * 30 days)
### Cost Optimization (Spot Instances)
- Community Cloud (Spot): $0.13/hr (70% cheaper)
- TFT training: **$0.004** per run (73% savings)
- Batch training: **$0.011** per run (72% savings)
- Monthly cost (daily batch): **$0.33** (72% savings)
---
## Production Readiness Checklist
| Item | Status | Notes |
|------|--------|-------|
| ✅ Binaries compiled | **COMPLETE** | 5/5 FP32 models |
| ✅ CUDA support verified | **COMPLETE** | CUDA 12.9 linked |
| ✅ Binaries uploaded to S3 | **COMPLETE** | 85.9 MB total |
| ✅ Deployment manifest | **COMPLETE** | SHA-256 checksums |
| ✅ Test pass rate 100% | **COMPLETE** | 1,337/1,337 ML, 3,196/3,196 total |
| ✅ P0 bugs fixed | **COMPLETE** | 3/3 (TFT, MAMBA-2, PPO) |
| ⏳ Test data uploaded | **PENDING** | DEPLOY-02 |
| ⏳ Runpod pod template | **PENDING** | DEPLOY-03 |
| ⏳ Training validated | **PENDING** | DEPLOY-04 |
| ⏳ Checkpoints verified | **PENDING** | DEPLOY-05 |
**Overall Status**: 6/10 complete (60%), ready for next phase
---
## Key Resources
### Runpod Console
- **URL**: https://www.runpod.io/console/pods
- **Purpose**: Create/manage GPU pods
- **Required**: API key, payment method
### Runpod S3 Bucket
- **Bucket**: `s3://se3zdnb5o4/`
- **Region**: `eur-is-1`
- **Endpoint**: `https://s3api-eur-is-1.runpod.io`
- **Contents**:
- `binaries/` - 5 training binaries (85.9 MB)
- `test_data/` - Parquet data files (pending DEPLOY-02)
- `models/` - Trained model checkpoints (pending DEPLOY-04)
- `runpod_deployment_manifest.json` - Deployment metadata
### AWS CLI Profile
- **Profile name**: `runpod`
- **Region**: `eur-is-1`
- **Verify**: `aws configure list-profiles | grep runpod`
---
## Troubleshooting
### Quick Diagnostics
```bash
# Verify AWS profile
aws configure list-profiles | grep runpod
# List binaries on S3
aws s3 ls s3://se3zdnb5o4/binaries/ \
--profile runpod \
--endpoint-url https://s3api-eur-is-1.runpod.io
# Download manifest
aws s3 cp s3://se3zdnb5o4/runpod_deployment_manifest.json /tmp/manifest.json \
--profile runpod \
--endpoint-url https://s3api-eur-is-1.runpod.io
# Verify checksums
cat /tmp/manifest.json | jq -r '.binaries[] | "\(.name): \(.sha256)"'
```
### Common Issues
See **RUNPOD_DEPLOYMENT_COMMANDS.md** (Troubleshooting section) for:
- Binary download fails (AWS credentials)
- Out of GPU memory (use RTX 4090 or gradient checkpointing)
- Test data not found (upload test_data/*.parquet first)
- Checkpoint not saved (create /workspace/models directory)
---
## Next Steps
### Immediate (DEPLOY-02): Upload Test Data
**Estimated Time**: 10 minutes
**Command**:
```bash
aws s3 cp test_data/ES_FUT_180d.parquet s3://se3zdnb5o4/test_data/ \
--profile runpod --endpoint-url https://s3api-eur-is-1.runpod.io
# Repeat for NQ_FUT, 6E_FUT, ZN_FUT
```
### Short-Term (DEPLOY-03): Create Pod Template
**Estimated Time**: 15 minutes
**See**: RUNPOD_DEPLOYMENT_COMMANDS.md (Option 1)
### Short-Term (DEPLOY-04): Run TFT Training
**Estimated Time**: 5 minutes (2 min training + 3 min setup)
**Expected Cost**: $0.015
### Short-Term (DEPLOY-05): Validate Checkpoints
**Estimated Time**: 10 minutes
**Command**:
```bash
# Download checkpoint
aws s3 cp s3://se3zdnb5o4/models/tft_final_*.safetensors \
./models/tft_final.safetensors \
--profile runpod --endpoint-url https://s3api-eur-is-1.runpod.io
# Run inference test
cargo run -p ml --example test_tft_inference --release -- \
--checkpoint ./models/tft_final.safetensors \
--test-file test_data/ES_FUT_small.parquet
```
---
## Summary
✅ **DEPLOY-01 COMPLETE**: All 5 FP32 training binaries compiled, verified, and uploaded to Runpod S3 (85.9 MB total). Deployment manifest created with SHA-256 checksums. Production certified with 100% test pass rate (1,337/1,337 ML tests, 3,196/3,196 workspace tests). Ready for immediate GPU deployment with zero blockers.
**Next Agent**: DEPLOY-02 (Upload test data to Runpod S3)
**Files to Read**:
1. **Quick start**: RUNPOD_DEPLOYMENT_COMMANDS.md
2. **Full details**: AGENT_DEPLOY_01_RUNPOD_UPLOAD.md
3. **Status check**: DEPLOY_01_STATUS.txt
---
**End of Index**
Reference in New Issue View Git Blame Copy Permalink