gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
Files
46fea9a0e39fe82db8a7eda4cc43bdfd92cc299a
foxhunt/docs/archive/wave_d/reports/S3_QUICK_REFERENCE.md
jgrusewski 433af5c25d chore: Major codebase cleanup - remove deprecated files and organize structure 
- Docker: Delete 23 deprecated Dockerfiles, fix CI/CD to use Dockerfile.foxhunt-build
- Config: Remove 36 .env files, keep 4 essential, delete config/environments/
- Docs: Archive 614 Wave D files to docs/archive/wave_d/, 95% reduction in root
- Scripts: Delete 56 deprecated scripts, keep 58 production-critical (49% reduction)
- Python: Organize 37 scripts into scripts/python/ subdirectories, delete ml/python/
- Build: Remove 1GB artifacts, delete old venvs, clean Python cache from git
- Migrations: Delete deprecated directory (4,432 lines), remove duplicate database/migrations/
- Infrastructure: Delete deployment/ (61 files), docs/scripts/ (8 files)

Total impact: ~2,500 files cleaned, 750MB+ space freed, zero production impact
All deleted scripts backed up to archives. runpod/ and tests/runpod/ preserved.
data_acquisition_service retained per user request.
2025-10-30 01:02:34 +01:00

4.8 KiB
Raw Blame History

S3 Directory Structure - Quick Reference

Bucket: se3zdnb5o4
Endpoint: https://s3api-eur-is-1.runpod.io
Profile: runpod

Production Binaries

# List current binaries
aws s3 ls s3://se3zdnb5o4/binaries/current/ --profile runpod --endpoint-url https://s3api-eur-is-1.runpod.io

# Download specific binary
aws s3 cp s3://se3zdnb5o4/binaries/current/train_tft_parquet . --profile runpod --endpoint-url https://s3api-eur-is-1.runpod.io

# Upload new binary
aws s3 cp ./my_binary s3://se3zdnb5o4/binaries/current/my_binary --profile runpod --endpoint-url https://s3api-eur-is-1.runpod.io

Available binaries:

  • hyperopt_mamba2_demo (21.3 MB)
  • train_dqn (17.9 MB)
  • train_mamba2_parquet (17.8 MB)
  • train_ppo (11.4 MB)
  • train_tft_parquet (18.6 MB)

Datasets

Futures (Parquet)

# List futures datasets
aws s3 ls s3://se3zdnb5o4/datasets/parquet/futures/ --profile runpod --endpoint-url https://s3api-eur-is-1.runpod.io

# Download ES futures
aws s3 cp s3://se3zdnb5o4/datasets/parquet/futures/ES_FUT_180d.parquet . --profile runpod --endpoint-url https://s3api-eur-is-1.runpod.io

Available datasets:

  • 6E_FUT_180d.parquet (2.9 MB) - Euro futures 180 days
  • ES_FUT_180d.parquet (3.0 MB) - E-mini S&P 500 180 days
  • NQ_FUT_180d.parquet (4.6 MB) - NASDAQ 100 180 days
  • ZN_FUT_90d.parquet (2.8 MB) - 10-Year T-Note 90 days
  • (+ 5 small test files)

Crypto (Parquet)

# List crypto datasets
aws s3 ls s3://se3zdnb5o4/datasets/parquet/crypto/ --profile runpod --endpoint-url https://s3api-eur-is-1.runpod.io

# Download BTC dataset
aws s3 cp s3://se3zdnb5o4/datasets/parquet/crypto/BTC-USD_30day_2024-09.parquet . --profile runpod --endpoint-url https://s3api-eur-is-1.runpod.io

Available datasets:

  • BTC-USD_30day_2024-09.parquet (891 KB) - Bitcoin 30 days
  • ETH-USD_30day_2024-09.parquet (819 KB) - Ethereum 30 days

Training Outputs

# Upload training output (TFT example)
aws s3 cp ./tft_model.safetensors s3://se3zdnb5o4/training_runs/tft/tft_model_20251028.safetensors --profile runpod --endpoint-url https://s3api-eur-is-1.runpod.io

# List TFT training runs
aws s3 ls s3://se3zdnb5o4/training_runs/tft/ --profile runpod --endpoint-url https://s3api-eur-is-1.runpod.io --recursive

Available directories:

  • training_runs/mamba2/ - MAMBA-2 training outputs
  • training_runs/tft/ - TFT training outputs
  • training_runs/dqn/ - DQN training outputs
  • training_runs/ppo/ - PPO training outputs

Production Models

# Upload certified production model
aws s3 cp ./tft_production.safetensors s3://se3zdnb5o4/production_models/tft_v1.0.safetensors --profile runpod --endpoint-url https://s3api-eur-is-1.runpod.io

# List production models
aws s3 ls s3://se3zdnb5o4/production_models/ --profile runpod --endpoint-url https://s3api-eur-is-1.runpod.io --recursive

Runpod Volume Mount Paths

When mounted at /runpod-volume/:

# Binaries
/runpod-volume/binaries/current/train_tft_parquet

# Datasets
/runpod-volume/datasets/parquet/futures/ES_FUT_180d.parquet
/runpod-volume/datasets/parquet/crypto/BTC-USD_30day_2024-09.parquet

# Training outputs
/runpod-volume/training_runs/tft/

# Production models
/runpod-volume/production_models/

Sync Operations

# Sync all binaries to local
aws s3 sync s3://se3zdnb5o4/binaries/current/ ./binaries/ --profile runpod --endpoint-url https://s3api-eur-is-1.runpod.io

# Sync all futures datasets to local
aws s3 sync s3://se3zdnb5o4/datasets/parquet/futures/ ./datasets/futures/ --profile runpod --endpoint-url https://s3api-eur-is-1.runpod.io

# Upload training run (entire directory)
aws s3 sync ./training_output/ s3://se3zdnb5o4/training_runs/tft/run_20251028/ --profile runpod --endpoint-url https://s3api-eur-is-1.runpod.io

Common Tasks

Check file existence

aws s3 ls s3://se3zdnb5o4/binaries/current/train_tft_parquet --profile runpod --endpoint-url https://s3api-eur-is-1.runpod.io

Get file size

aws s3 ls s3://se3zdnb5o4/datasets/parquet/futures/ES_FUT_180d.parquet --profile runpod --endpoint-url https://s3api-eur-is-1.runpod.io --human-readable

List all files recursively

aws s3 ls s3://se3zdnb5o4/ --profile runpod --endpoint-url https://s3api-eur-is-1.runpod.io --recursive | grep "binaries/current/"

Delete file (use with caution)

aws s3 rm s3://se3zdnb5o4/path/to/file --profile runpod --endpoint-url https://s3api-eur-is-1.runpod.io

Archive Policy

  • binaries/current/: Latest production binaries only
  • binaries/archive/: Old versions (keep 3 most recent)
  • training_runs/: Keep last 10 runs per model
  • production_models/: Keep all certified models

Last Updated: 2025-10-28
Migration Report: See S3_MIGRATION_REPORT.md

Reference in New Issue View Git Blame Copy Permalink