foxhunt/docs/archive/wave_d/reports/RUNPOD_S3_INVENTORY.md

RunPod S3 Storage Inventory Report

Generated: 2025-10-25 00:55 UTC Volume ID: se3zdnb5o4 Region: EUR-IS-1 Endpoint: https://s3api-eur-is-1.runpod.io

---

Summary

Category	File Count	Total Size	Status
Binaries	5 files	80.2 MB	✅ Complete
Training Data	11 files	12.8 MB	✅ Complete
Trained Models	12 files	1.85 MB	✅ DQN Training Complete
Debug Tests	2 files	7.3 MB	✅ Present
Logs/Checkpoints	3 files	90 bytes	✅ Initialized
TOTAL	35 files	103.9 MB	✅ Operational

---

📂 Directory Structure

s3://se3zdnb5o4/
├── .env                                 (1.5 KB)
├── AGENT_5_S3_UPLOAD_REPORT.md         (8.0 KB)
├── binaries/                            [80.2 MB, 5 files]
│   ├── CHECKSUMS.txt                    (323 bytes)
│   ├── train_dqn                        (22.2 MB) ✅
│   ├── train_mamba2_parquet             (22.0 MB) ✅
│   ├── train_ppo                        (13.1 MB) ✅
│   └── train_tft_parquet                (22.9 MB) ✅
├── checkpoints/                         [30 bytes]
│   └── README.txt
├── debug_tests/                         [7.3 MB, 2 files]
│   ├── test1_hello                      (3.6 MB)
│   └── test2_cuda_check                 (3.7 MB)
├── logs/                                [26 bytes]
│   └── README.txt
├── models/                              [1.85 MB, 12 files] 🆕 DQN TRAINED
│   ├── README.txt                       (34 bytes)
│   ├── metadata/                        (empty)
│   ├── dqn_epoch_10.safetensors         (154.4 KB) ⏰ 2025-10-24 22:46
│   ├── dqn_epoch_20.safetensors         (154.4 KB) ⏰ 2025-10-24 22:47
│   ├── dqn_epoch_30.safetensors         (154.4 KB) ⏰ 2025-10-24 22:47
│   ├── dqn_epoch_40.safetensors         (154.4 KB) ⏰ 2025-10-24 22:47
│   ├── dqn_epoch_50.safetensors         (154.4 KB) ⏰ 2025-10-24 22:48
│   ├── dqn_epoch_60.safetensors         (154.4 KB) ⏰ 2025-10-24 22:44
│   ├── dqn_epoch_70.safetensors         (154.4 KB) ⏰ 2025-10-24 22:45
│   ├── dqn_epoch_80.safetensors         (154.4 KB) ⏰ 2025-10-24 22:45
│   ├── dqn_epoch_90.safetensors         (154.4 KB) ⏰ 2025-10-24 22:45
│   ├── dqn_epoch_100.safetensors        (154.4 KB) ⏰ 2025-10-24 22:46
│   ├── dqn_final_epoch1.safetensors     (154.4 KB) ⏰ 2025-10-25 00:53 🆕 LATEST
│   └── dqn_final_epoch100.safetensors   (154.4 KB) ⏰ 2025-10-24 22:46
└── test_data/                           [12.8 MB, 11 files]
    ├── 6E_FUT_180d.parquet              (2.7 MB) ✅
    ├── 6E_FUT_small.parquet             (22.3 KB)
    ├── ES_FUT_180d.parquet              (2.9 MB) ✅
    ├── ES_FUT_small.parquet             (24.7 KB)
    ├── NQ_FUT_180d.parquet              (4.3 MB) ✅
    ├── NQ_FUT_small.parquet             (26.6 KB)
    ├── ZN_FUT_90d.parquet               (2.7 MB) ✅
    ├── ZN_FUT_90d_clean.parquet         (64.6 KB)
    ├── ZN_FUT_small.parquet             (18.8 KB)
    └── real/parquet/
        ├── BTC-USD_30day_2024-09.parquet (871.0 KB)
        └── ETH-USD_30day_2024-09.parquet (800.3 KB)

---

✅ Binary Verification

All 4 training binaries present with verified checksums:

8412e3426ca7d53e2db18a0181656649f7398aff392d0889ed18c6d9e488a93f  train_dqn
8063275fb2db1252f879b5d7852680b140b2c41b56403a7e496a3a3ebed2b692  train_mamba2_parquet
e5b6b566c85ec83cd118332c986a78ca1fa1580781b5515b84b80a391f5c32da  train_ppo
47061c765ae8568da238d52dd93993ed9f4b0cfaf003206699a01047cbd22d52  train_tft_parquet

Status: ✅ All binaries uploaded successfully on 2025-10-24 (train_tft_parquet updated at 21:42)

---

🎯 Trained Model: DQN (Pod 9nixt6bhskpexb)

Latest Model: dqn_final_epoch1.safetensors (154.4 KB) Timestamp: 2025-10-25 00:53:57 UTC SHA256: 28f11850f7326a188c9bd280e9b4633a1961ae37d1d33ad0c46a8070e9b3ebb6 Download Verified: ✅ Successfully downloaded and verified (155 KB)

Training Checkpoints (11 files, 10-epoch intervals):

• Epoch 10, 20, 30, 40, 50, 60, 70, 80, 90, 100 ✅
• Final checkpoint: dqn_final_epoch1.safetensors (most recent) 🆕
• Final checkpoint: dqn_final_epoch100.safetensors (older training run)

Observations:

• Two training runs detected:
  • Run 1: 100 epochs (dqn_epoch_10 through dqn_epoch_100), completed 2025-10-24 22:48
  • Run 2: 1 epoch (dqn_final_epoch1.safetensors), completed 2025-10-25 00:53 🆕 LATEST
• All checkpoints are same size (154.4 KB), indicating consistent model architecture
• ⚠️ Timestamp anomaly: Epochs 60-90 have earlier timestamps (22:44-22:45) than epochs 10-50 (22:46-22:48)
  • Likely due to S3 async upload delays or clock skew

---

📊 Training Data Assets

Production Datasets (4 futures, 180 days):

Symbol	File	Size	Status
ES.FUT	ES_FUT_180d.parquet	2.9 MB	✅ Ready
NQ.FUT	NQ_FUT_180d.parquet	4.3 MB	✅ Ready
6E.FUT	6E_FUT_180d.parquet	2.7 MB	✅ Ready
ZN.FUT	ZN_FUT_90d.parquet	2.7 MB	✅ Ready (90 days)

Test Datasets (small samples for smoke tests):

• ES_FUT_small.parquet (24.7 KB)
• NQ_FUT_small.parquet (26.6 KB)
• 6E_FUT_small.parquet (22.3 KB)
• ZN_FUT_small.parquet (18.8 KB)
• ZN_FUT_90d_clean.parquet (64.6 KB)

Crypto Data (real market data):

• BTC-USD_30day_2024-09.parquet (871.0 KB)
• ETH-USD_30day_2024-09.parquet (800.3 KB)

Coverage:

• ✅ All 4 primary futures symbols (ES, NQ, 6E, ZN) have 90-180 day datasets
• ✅ Small test files for rapid smoke testing
• ✅ Real crypto data for alternative asset testing

---

🔍 Key Findings

✅ Successes

1. All binaries uploaded: 4/4 training binaries present (80.2 MB total)
2. DQN training complete: 12 model checkpoints saved to S3
3. Latest model available: dqn_final_epoch1.safetensors (2025-10-25 00:53)
4. Training data complete: All 4 futures + 2 crypto symbols present
5. Checksums verified: All binary checksums match upload records
6. Volume healthy: 35 files, 103.9 MB total (well below 50 GB limit)

⚠️ Observations

1. Two DQN training runs: 100-epoch run (completed) + 1-epoch run (latest)
2. Timestamp anomalies: S3 upload timestamps not strictly sequential (likely async upload)
3. Metadata directory empty: /models/metadata/ has no files yet
4. README mismatch: /models/README.txt says "TFT model checkpoints" but contains DQN models

🎯 Next Training Targets (Not Yet Present)

• ❌ PPO models (train_ppo binary ready, no models yet)
• ❌ MAMBA-2 models (train_mamba2_parquet binary ready, no models yet)
• ❌ TFT models (train_tft_parquet binary ready, no models yet)

---

💡 Recommendations

1. Immediate Actions:

   • ✅ DQN model can be downloaded and integrated into trading system
   • ✅ Verify model performance metrics from pod logs (Sharpe, win rate, etc.)
   • ⏳ Update /models/README.txt to reflect DQN models (currently says "TFT")

2. Next Training Steps:

   • Train PPO model (binary ready, est. 7-10s training time)
   • Train MAMBA-2 model (binary ready, est. 2-3 min training time)
   • Train TFT model (binary ready, est. 3-5 min training time)

3. Storage Optimization:

   • Current usage: 103.9 MB / 50 GB (0.2%)
   • Headroom: 49.9 GB available for future models
   • Cost: $5/month (volume) + ~$0.01/training run

4. Monitoring:

   • Track S3 upload timestamps for anomalies
   • Implement model validation checksums
   • Set up automated model registry (track model metadata)

---

📁 File Access Examples

Download latest DQN model:

aws s3 cp s3://se3zdnb5o4/models/dqn_final_epoch1.safetensors . \
  --profile runpod \
  --endpoint-url https://s3api-eur-is-1.runpod.io

List all models:

aws s3 ls s3://se3zdnb5o4/models/ \
  --profile runpod \
  --endpoint-url https://s3api-eur-is-1.runpod.io \
  --recursive --human-readable

Upload new training data:

aws s3 cp new_data.parquet s3://se3zdnb5o4/test_data/ \
  --profile runpod \
  --endpoint-url https://s3api-eur-is-1.runpod.io

---

🔐 Security Status

• ✅ AWS CLI profile configured with RunPod credentials
• ✅ Private S3-compatible storage (not publicly accessible)
• ✅ Credentials stored in .env.runpod (gitignored)
• ✅ Volume mounted read-write in pods for training persistence

---

Report Generated: 2025-10-25 00:55 UTC Next Review: After PPO/MAMBA-2/TFT training runs