# Production S3 Organization Structure for ML Training **Created**: 2025-10-28 **Bucket**: `s3://se3zdnb5o4` (Runpod EUR-IS-1) **Status**: Design Complete - Ready for Implementation --- ## Executive Summary This document defines a production-ready S3 organization structure for ML training on Runpod. The design scales to hundreds of training runs, supports hyperparameter optimization metadata, and enables efficient retrieval of models and results. **Key Features**: - Versioned binaries with timestamps - Hierarchical training run organization - Rich metadata capture (hyperparams, git commit, metrics) - Hyperopt trial tracking with Optuna integration - Production model promotion workflow - Zero-downtime migration from current flat structure --- ## 1. Current S3 Structure Analysis ### Current Organization (Flat, Unstructured) ``` s3://se3zdnb5o4/ ├── .env (1.5 KB) ├── AGENT_5_S3_UPLOAD_REPORT.md (8.0 KB) ├── binaries/ [80.2 MB, 5 files] │ ├── CHECKSUMS.txt │ ├── hyperopt_mamba2_demo (21 MB, multiple versions) │ ├── hyperopt_mamba2_demo_CUDA12 (duplicates) │ ├── hyperopt_mamba2_demo_CUDA12_20251027_234229 (timestamped) │ ├── hyperopt_mamba2_demo_FIXED (more duplicates) │ ├── train_dqn (17.8 MB) │ ├── train_mamba2_dbn (13.9 MB) │ ├── train_mamba2_parquet (17.8 MB) │ ├── train_mamba2_parquet_ADAMW_FIX (20.7 MB) │ ├── train_ppo (11.4 MB) │ ├── train_ppo_parquet (20.7 MB) │ ├── train_tft_parquet (18.5 MB) │ └── (multiple checksum files) ├── checkpoints/ [30 bytes] │ └── README.txt ├── debug_tests/ [7.3 MB] │ ├── test1_hello │ └── test2_cuda_check ├── logs/ [26 bytes] │ └── README.txt ├── models/ [1.85 MB, 12 files] - FLAT! │ ├── dqn_epoch_10.safetensors │ ├── dqn_epoch_20.safetensors │ ├── ... (epochs 30-90) │ ├── dqn_epoch_100.safetensors │ ├── dqn_final_epoch1.safetensors │ ├── dqn_final_epoch100.safetensors │ └── tft_180d_50ep_rtx4090_bs64/ │ ├── tft_225_epoch_0.json │ └── tft_225_epoch_0.safetensors └── test_data/ [12.8 MB] ├── (9 parquet files - unorganized) └── real/ ├── databento/ml_training/ (392 .dbn files) └── parquet/ (2 crypto files) ``` ### Issues Identified 1. **Binary Versioning Chaos**: 5+ versions of `hyperopt_mamba2_demo` with unclear naming 2. **No Training Run Organization**: All DQN checkpoints in flat `/models/` directory 3. **Missing Metadata**: No run ID, hyperparameters, git commit, or trial info 4. **No Hyperopt Tracking**: Can't track Optuna trials or best parameters 5. **Unclear Production Status**: No way to identify "promoted" production models 6. **Data Duplication**: Multiple dataset locations (`test_data/`, `test_data/real/`) 7. **No Logs**: Empty `/logs/` directory - training logs not captured 8. **Checksum Files Everywhere**: 4 different `CHECKSUMS.txt` files in `/binaries/` --- ## 2. Production S3 Directory Structure ### Complete Hierarchy ``` s3://se3zdnb5o4/ ├── binaries/ # Versioned training binaries │ ├── current/ # Latest stable versions (symlinks) │ │ ├── train_dqn -> ../v20251028_134530/train_dqn │ │ ├── train_mamba2_parquet -> ../v20251027_225930/train_mamba2_parquet │ │ ├── train_ppo -> ../v20251028_134530/train_ppo │ │ ├── train_tft_parquet -> ../v20251028_134530/train_tft_parquet │ │ └── hyperopt_mamba2_demo -> ../v20251028_224635/hyperopt_mamba2_demo │ ├── v20251028_134530/ # Version: YYYYMMDD_HHMMSS │ │ ├── train_dqn (17.8 MB) │ │ ├── train_ppo (11.4 MB) │ │ ├── train_tft_parquet (18.5 MB) │ │ ├── checksums.sha256 # SHA256 hashes for this version │ │ └── build_metadata.json # Git commit, build time, Rust version │ ├── v20251028_224635/ │ │ ├── hyperopt_mamba2_demo (21 MB) │ │ ├── checksums.sha256 │ │ └── build_metadata.json │ └── v20251027_225930/ │ ├── train_mamba2_parquet (17.8 MB) │ ├── checksums.sha256 │ └── build_metadata.json │ ├── datasets/ # Organized training data │ ├── parquet/ # Parquet format (preferred) │ │ ├── futures/ │ │ │ ├── ES_FUT_180d.parquet (2.9 MB) │ │ │ ├── NQ_FUT_180d.parquet (4.3 MB) │ │ │ ├── 6E_FUT_180d.parquet (2.7 MB) │ │ │ └── ZN_FUT_90d.parquet (2.7 MB) │ │ ├── futures_small/ # Smoke test datasets │ │ │ ├── ES_FUT_small.parquet (24.7 KB) │ │ │ ├── NQ_FUT_small.parquet (26.6 KB) │ │ │ ├── 6E_FUT_small.parquet (22.3 KB) │ │ │ └── ZN_FUT_small.parquet (18.8 KB) │ │ └── crypto/ │ │ ├── BTC-USD_30day_2024-09.parquet (871 KB) │ │ └── ETH-USD_30day_2024-09.parquet (800 KB) │ └── dbn/ # Databento format (historical) │ └── ohlcv-1m/ │ ├── ES.FUT/ │ │ └── (98 .dbn files) │ ├── NQ.FUT/ │ │ └── (98 .dbn files) │ ├── 6E.FUT/ │ │ └── (98 .dbn files) │ └── ZN.FUT/ │ └── (98 .dbn files) │ ├── training_runs/ # All training runs (hyperopt + manual) │ ├── mamba2/ │ │ ├── run_20251028_224530_hyperopt/ # Hyperopt run │ │ │ ├── metadata.json # Run config, git commit, dataset │ │ │ ├── hyperopt/ │ │ │ │ ├── study.db # Optuna SQLite database │ │ │ │ ├── trial_history.json # All trials (params + metrics) │ │ │ │ ├── best_params.json # Best hyperparameters found │ │ │ │ └── optuna_visualization.html # Plots (optional) │ │ │ ├── checkpoints/ │ │ │ │ ├── trial_0_epoch_5.safetensors │ │ │ │ ├── trial_0_epoch_10.safetensors │ │ │ │ ├── trial_1_epoch_5.safetensors │ │ │ │ ├── ... │ │ │ │ └── best_model.safetensors # Best overall model │ │ │ ├── logs/ │ │ │ │ ├── hyperopt.log # Hyperopt progress │ │ │ │ ├── trial_0.log # Individual trial logs │ │ │ │ ├── trial_1.log │ │ │ │ └── ... │ │ │ └── metrics/ │ │ │ ├── training_curves.png # Loss/accuracy plots │ │ │ ├── param_importance.png # Hyperparameter sensitivity │ │ │ └── convergence.png # Optimization convergence │ │ ├── run_20251028_153045_manual/ # Manual training run │ │ │ ├── metadata.json │ │ │ ├── checkpoints/ │ │ │ │ ├── epoch_5.safetensors │ │ │ │ ├── epoch_10.safetensors │ │ │ │ ├── ... │ │ │ │ └── final_model.safetensors │ │ │ ├── logs/ │ │ │ │ └── training.log │ │ │ └── metrics/ │ │ │ ├── training_losses.csv │ │ │ └── training_metrics.json │ │ └── run_20251027_180000_baseline/ # Baseline comparison run │ │ └── ... │ ├── tft/ │ │ ├── run_20251028_100000_hyperopt/ │ │ │ └── ... (same structure) │ │ └── run_20251028_120000_manual/ │ │ └── ... │ ├── dqn/ │ │ ├── run_20251025_005300_manual/ # Existing DQN run (migrated) │ │ │ ├── metadata.json │ │ │ ├── checkpoints/ │ │ │ │ ├── epoch_10.safetensors │ │ │ │ ├── epoch_20.safetensors │ │ │ │ ├── ... │ │ │ │ └── final_epoch_100.safetensors │ │ │ ├── logs/ │ │ │ │ └── training.log (empty - not captured) │ │ │ └── metrics/ │ │ │ └── (empty - not captured) │ │ └── run_20251028_140000_hyperopt/ │ │ └── ... │ └── ppo/ │ ├── run_20251028_160000_hyperopt/ │ │ └── ... │ └── run_20251028_170000_manual/ │ └── ... │ └── production_models/ # Promoted production-ready models ├── mamba2/ │ ├── v1_20251028_224530/ # Production version 1 │ │ ├── model.safetensors # Promoted from training_runs/ │ │ ├── metadata.json # Copy of training metadata │ │ ├── validation_report.json # Production validation results │ │ └── promotion_notes.md # Why this model was promoted │ └── latest -> v1_20251028_224530 # Symlink to latest version ├── tft/ │ ├── v1_20251028_120000/ │ │ └── ... │ └── latest -> v1_20251028_120000 ├── dqn/ │ ├── v1_20251025_005300/ │ │ └── ... │ └── latest -> v1_20251025_005300 └── ppo/ ├── v1_20251028_170000/ │ └── ... └── latest -> v1_20251028_170000 ``` --- ## 3. Metadata Schemas ### 3.1 Training Run Metadata (`metadata.json`) Captures complete training context for reproducibility. ```json { "run_id": "run_20251028_224530_hyperopt", "model_type": "mamba2", "run_type": "hyperopt", "created_at": "2025-10-28T22:45:30Z", "git_commit": "3f4aae20", "git_branch": "main", "git_dirty": false, "binary_version": "v20251028_134530", "binary_checksum": "8063275fb2db1252f879b5d7852680b140b2c41b56403a7e496a3a3ebed2b692", "dataset": { "path": "s3://se3zdnb5o4/datasets/parquet/futures/ES_FUT_180d.parquet", "symbol": "ES.FUT", "timeframe": "1m", "bars": 259200, "date_range": ["2024-01-02", "2024-06-30"], "checksum": "sha256:..." }, "hardware": { "gpu_type": "RTX A4000", "vram_gb": 16, "cuda_version": "12.9.1", "driver_version": "550.54.15" }, "training_config": { "epochs": 50, "batch_size": 32, "learning_rate": 0.0001, "sequence_length": 60, "train_split": 0.8, "early_stopping_patience": 20 }, "hyperparameters": { "d_model": 225, "d_state": 16, "num_layers": 6, "dropout": 0.1, "weight_decay": 0.0001, "grad_clip": 1.0, "warmup_steps": 1000, "adam_beta1": 0.9, "adam_beta2": 0.999, "adam_epsilon": 1e-8, "lookback_window": 60, "sequence_stride": 1, "norm_eps": 1e-5 }, "training_duration": { "wall_time_seconds": 1860, "epochs_completed": 50, "best_epoch": 42 }, "final_metrics": { "train_loss": 0.003245, "val_loss": 0.003892, "perplexity": 1.0039, "directional_accuracy": 0.62, "mae": 0.0042, "rmse": 0.0068, "r_squared": 0.84 }, "tags": ["production_candidate", "rtx_a4000_optimized"] } ``` ### 3.2 Hyperopt Trial History (`trial_history.json`) Optuna-compatible trial tracking for analysis. ```json { "study_name": "mamba2_hyperopt_20251028_224530", "created_at": "2025-10-28T22:45:30Z", "total_trials": 30, "best_trial_number": 18, "best_value": 0.003892, "optimization_direction": "minimize", "search_space": { "learning_rate": {"type": "log_uniform", "low": 1e-5, "high": 1e-2}, "batch_size": {"type": "int", "low": 4, "high": 96}, "dropout": {"type": "uniform", "low": 0.0, "high": 0.5}, "weight_decay": {"type": "log_uniform", "low": 1e-6, "high": 1e-2}, "grad_clip": {"type": "log_uniform", "low": 0.5, "high": 5.0}, "warmup_steps": {"type": "int", "low": 100, "high": 2000}, "adam_beta1": {"type": "uniform", "low": 0.85, "high": 0.95}, "adam_beta2": {"type": "uniform", "low": 0.98, "high": 0.999}, "adam_epsilon": {"type": "log_uniform", "low": 1e-9, "high": 1e-7}, "lookback_window": {"type": "int", "low": 30, "high": 120}, "sequence_stride": {"type": "int", "low": 1, "high": 5}, "norm_eps": {"type": "log_uniform", "low": 1e-6, "high": 1e-4} }, "trials": [ { "trial_number": 0, "state": "COMPLETE", "value": 0.008234, "datetime_start": "2025-10-28T22:45:35Z", "datetime_complete": "2025-10-28T22:47:52Z", "duration_seconds": 137, "params": { "learning_rate": 0.000123, "batch_size": 32, "dropout": 0.15, "weight_decay": 0.00015, "grad_clip": 1.2, "warmup_steps": 500, "adam_beta1": 0.9, "adam_beta2": 0.999, "adam_epsilon": 1e-8, "lookback_window": 60, "sequence_stride": 1, "norm_eps": 1e-5 }, "user_attrs": { "train_loss": 0.007891, "epochs_completed": 50, "gpu_memory_peak_mb": 1420 } }, { "trial_number": 1, "state": "COMPLETE", "value": 0.006123, "datetime_start": "2025-10-28T22:47:55Z", "datetime_complete": "2025-10-28T22:50:10Z", "duration_seconds": 135, "params": { "...": "..." }, "user_attrs": { "...": "..." } } ] } ``` ### 3.3 Best Hyperparameters (`best_params.json`) Quick reference for production deployment. ```json { "study_name": "mamba2_hyperopt_20251028_224530", "best_trial_number": 18, "best_value": 0.003892, "found_at": "2025-10-28T23:32:15Z", "params": { "learning_rate": 0.000087, "batch_size": 48, "dropout": 0.12, "weight_decay": 0.00012, "grad_clip": 1.5, "warmup_steps": 800, "adam_beta1": 0.91, "adam_beta2": 0.9995, "adam_epsilon": 2.3e-8, "lookback_window": 75, "sequence_stride": 2, "norm_eps": 3.5e-6 }, "metrics": { "train_loss": 0.003421, "val_loss": 0.003892, "directional_accuracy": 0.64, "mae": 0.0038 }, "checkpoint_path": "s3://se3zdnb5o4/training_runs/mamba2/run_20251028_224530_hyperopt/checkpoints/best_model.safetensors", "recommended_for_production": true, "notes": "Best model found after 18 trials. Significantly better directional accuracy (64% vs 58% baseline)." } ``` ### 3.4 Binary Build Metadata (`build_metadata.json`) Tracks binary provenance for debugging. ```json { "version": "v20251028_134530", "build_timestamp": "2025-10-28T13:45:30Z", "git_commit": "3f4aae20", "git_branch": "main", "git_dirty": false, "git_remote": "https://github.com/foxhunt/ml.git", "rust_version": "1.75.0", "cargo_profile": "release", "features": ["cuda", "mimalloc-allocator"], "target_triple": "x86_64-unknown-linux-gnu", "cuda_version": "12.9.1", "binaries": { "train_dqn": { "size_bytes": 17875296, "sha256": "8412e3426ca7d53e2db18a0181656649f7398aff392d0889ed18c6d9e488a93f" }, "train_ppo": { "size_bytes": 11440200, "sha256": "e5b6b566c85ec83cd118332c986a78ca1fa1580781b5515b84b80a391f5c32da" }, "train_tft_parquet": { "size_bytes": 18578960, "sha256": "47061c765ae8568da238d52dd93993ed9f4b0cfaf003206699a01047cbd22d52" } }, "build_host": { "hostname": "build-server-1", "os": "Linux 6.14.0-33-generic", "arch": "x86_64" } } ``` ### 3.5 Production Promotion Metadata (`validation_report.json`) Justifies production deployment. ```json { "model_id": "mamba2_v1_20251028_224530", "promoted_from": "s3://se3zdnb5o4/training_runs/mamba2/run_20251028_224530_hyperopt/checkpoints/best_model.safetensors", "promoted_at": "2025-10-29T10:15:00Z", "promoted_by": "trading_team", "validation_results": { "holdout_dataset": "ES_FUT_2024-07-01_to_2024-09-30", "holdout_metrics": { "val_loss": 0.004012, "directional_accuracy": 0.63, "sharpe_ratio": 2.15, "max_drawdown": 0.12, "win_rate": 0.61 }, "backtesting": { "period": "2024-07-01 to 2024-09-30", "initial_capital": 100000, "final_equity": 128450, "total_return": 0.2845, "sharpe_ratio": 2.15, "max_drawdown": 0.12, "win_rate": 0.61, "profit_factor": 1.85 } }, "comparison_to_baseline": { "baseline_model": "mamba2_default_params", "improvement_directional_accuracy": "+8.5%", "improvement_sharpe": "+0.35", "improvement_drawdown": "-3.2%" }, "production_notes": "Significantly improved directional accuracy over baseline. Ready for 10% live capital allocation.", "rollback_plan": "Revert to mamba2_v0_baseline if Sharpe < 1.5 over 7 days", "approval": { "approved_by": "risk_committee", "approval_date": "2025-10-29T09:00:00Z", "signatures": ["john_doe", "jane_smith"] } } ``` --- ## 4. Migration Plan ### Phase 1: Create New Structure (Zero Downtime) **Duration**: 5 minutes **Risk**: Low (no deletions, only additions) ```bash # Create new directory structure aws s3api put-object --bucket se3zdnb5o4 \ --key binaries/current/.keep \ --profile runpod --endpoint-url https://s3api-eur-is-1.runpod.io aws s3api put-object --bucket se3zdnb5o4 \ --key datasets/parquet/futures/.keep \ --profile runpod --endpoint-url https://s3api-eur-is-1.runpod.io aws s3api put-object --bucket se3zdnb5o4 \ --key training_runs/mamba2/.keep \ --profile runpod --endpoint-url https://s3api-eur-is-1.runpod.io aws s3api put-object --bucket se3zdnb5o4 \ --key production_models/mamba2/.keep \ --profile runpod --endpoint-url https://s3api-eur-is-1.runpod.io # (Repeat for all models: tft, dqn, ppo) ``` ### Phase 2: Migrate Binaries (Keep Old Paths) **Duration**: 10 minutes **Risk**: Low (copies, not moves) ```bash # Copy current binaries to versioned structure VERSION="v20251028_$(date +%H%M%S)" aws s3 cp s3://se3zdnb5o4/binaries/train_dqn \ s3://se3zdnb5o4/binaries/$VERSION/train_dqn \ --profile runpod --endpoint-url https://s3api-eur-is-1.runpod.io # Create checksums sha256sum train_dqn > checksums.sha256 aws s3 cp checksums.sha256 \ s3://se3zdnb5o4/binaries/$VERSION/checksums.sha256 \ --profile runpod --endpoint-url https://s3api-eur-is-1.runpod.io # Create symlink metadata (S3 doesn't support symlinks, use JSON pointer) echo '{"target": "'$VERSION'/train_dqn"}' > current_train_dqn.json aws s3 cp current_train_dqn.json \ s3://se3zdnb5o4/binaries/current/train_dqn.json \ --profile runpod --endpoint-url https://s3api-eur-is-1.runpod.io # Repeat for all binaries ``` ### Phase 3: Migrate Datasets (Reorganize) **Duration**: 5 minutes **Risk**: Low (copies) ```bash # Move parquet files to organized structure aws s3 cp s3://se3zdnb5o4/test_data/ES_FUT_180d.parquet \ s3://se3zdnb5o4/datasets/parquet/futures/ES_FUT_180d.parquet \ --profile runpod --endpoint-url https://s3api-eur-is-1.runpod.io aws s3 cp s3://se3zdnb5o4/test_data/ES_FUT_small.parquet \ s3://se3zdnb5o4/datasets/parquet/futures_small/ES_FUT_small.parquet \ --profile runpod --endpoint-url https://s3api-eur-is-1.runpod.io # Repeat for all datasets (NQ, 6E, ZN, crypto) # Keep old paths as copies for backward compatibility # (Don't delete /test_data/ until all code is updated) ``` ### Phase 4: Migrate Existing Training Run (DQN) **Duration**: 5 minutes **Risk**: Low (copies) ```bash # Create metadata for existing DQN run RUN_DIR="s3://se3zdnb5o4/training_runs/dqn/run_20251025_005300_manual" # Generate metadata.json (from pod logs if available) cat > metadata.json << 'EOF' { "run_id": "run_20251025_005300_manual", "model_type": "dqn", "run_type": "manual", "created_at": "2025-10-25T00:53:00Z", "git_commit": "unknown", "dataset": { "path": "s3://se3zdnb5o4/test_data/ES_FUT_180d.parquet", "symbol": "ES.FUT" }, "training_config": { "epochs": 100 }, "notes": "Migrated from flat /models/ structure. Original training logs not captured." } EOF aws s3 cp metadata.json $RUN_DIR/metadata.json \ --profile runpod --endpoint-url https://s3api-eur-is-1.runpod.io # Copy checkpoints aws s3 cp s3://se3zdnb5o4/models/dqn_epoch_10.safetensors \ $RUN_DIR/checkpoints/epoch_10.safetensors \ --profile runpod --endpoint-url https://s3api-eur-is-1.runpod.io # Repeat for all epochs (10, 20, ..., 100) # Copy final model aws s3 cp s3://se3zdnb5o4/models/dqn_final_epoch100.safetensors \ $RUN_DIR/checkpoints/final_epoch_100.safetensors \ --profile runpod --endpoint-url https://s3api-eur-is-1.runpod.io # Keep old paths for now (backward compatibility) ``` ### Phase 5: Update Code to Use New Paths **Duration**: 2 hours **Risk**: Medium (requires testing) See **Section 5: Code Changes** below. ### Phase 6: Cleanup Old Structure (After Validation) **Duration**: 10 minutes **Risk**: Medium (irreversible deletions) ```bash # ONLY after confirming new structure works! # Delete old flat structures aws s3 rm s3://se3zdnb5o4/models/ --recursive \ --profile runpod --endpoint-url https://s3api-eur-is-1.runpod.io aws s3 rm s3://se3zdnb5o4/binaries/train_dqn \ --profile runpod --endpoint-url https://s3api-eur-is-1.runpod.io # (Keep old /test_data/ for backward compatibility until all code updated) ``` --- ## 5. Code Changes Required ### Files Requiring Path Updates #### 5.1 Hyperopt Adapters **Files**: - `/home/jgrusewski/Work/foxhunt/ml/src/hyperopt/adapters/mamba2.rs` - `/home/jgrusewski/Work/foxhunt/ml/src/hyperopt/adapters/tft.rs` - `/home/jgrusewski/Work/foxhunt/ml/src/hyperopt/adapters/dqn.rs` - `/home/jgrusewski/Work/foxhunt/ml/src/hyperopt/adapters/ppo.rs` **Changes**: ```rust // OLD: Flat checkpoint directory self.checkpoint_dir = PathBuf::from("/runpod-volume/checkpoints/mamba2_hyperopt"); // NEW: Timestamped run directory with hyperopt subdirectory let run_id = format!("run_{}_hyperopt", chrono::Utc::now().format("%Y%m%d_%H%M%S")); self.checkpoint_dir = PathBuf::from(format!("/runpod-volume/training_runs/mamba2/{}/checkpoints", run_id)); // Also save metadata self.metadata_path = PathBuf::from(format!("/runpod-volume/training_runs/mamba2/{}/metadata.json", run_id)); self.hyperopt_dir = PathBuf::from(format!("/runpod-volume/training_runs/mamba2/{}/hyperopt", run_id)); self.logs_dir = PathBuf::from(format!("/runpod-volume/training_runs/mamba2/{}/logs", run_id)); ``` #### 5.2 Training Examples **Files**: - `/home/jgrusewski/Work/foxhunt/ml/examples/train_mamba2_parquet.rs` - `/home/jgrusewski/Work/foxhunt/ml/examples/train_tft_parquet.rs` - `/home/jgrusewski/Work/foxhunt/ml/examples/train_dqn.rs` - `/home/jgrusewski/Work/foxhunt/ml/examples/train_ppo.rs` **Changes**: ```rust // OLD: Flat checkpoint directory checkpoint_dir: PathBuf::from("ml/checkpoints/mamba2_parquet"), // NEW: Timestamped run directory let run_id = format!("run_{}_manual", chrono::Utc::now().format("%Y%m%d_%H%M%S")); checkpoint_dir: PathBuf::from(format!("/runpod-volume/training_runs/mamba2/{}", run_id)), // OLD: Hardcoded dataset path parquet_file: PathBuf::from("test_data/ES_FUT_180d.parquet"), // NEW: Organized dataset path parquet_file: PathBuf::from("/runpod-volume/datasets/parquet/futures/ES_FUT_180d.parquet"), ``` #### 5.3 Runpod Deployment Script **File**: `/home/jgrusewski/Work/foxhunt/scripts/runpod_deploy.py` **Changes**: ```python # OLD: Flat binary path '--command', default='/runpod-volume/binaries/train_dqn ...' # NEW: Current symlink (points to latest version) '--command', default='/runpod-volume/binaries/current/train_dqn ...' # NEW: Use organized dataset paths --parquet-file /runpod-volume/datasets/parquet/futures/ES_FUT_180d.parquet # NEW: Output to timestamped run directory --output-dir /runpod-volume/training_runs/dqn/run_$(date +%Y%m%d_%H%M%S)_manual ``` #### 5.4 Hyperopt Examples **Files**: - `/home/jgrusewski/Work/foxhunt/ml/examples/hyperopt_mamba2_demo.rs` - `/home/jgrusewski/Work/foxhunt/ml/examples/hyperopt_tft_demo.rs` - `/home/jgrusewski/Work/foxhunt/ml/examples/hyperopt_dqn_demo.rs` - `/home/jgrusewski/Work/foxhunt/ml/examples/hyperopt_ppo_demo.rs` **Changes**: ```rust // OLD: Uses default checkpoint_dir from adapter let trainer = Mamba2Trainer::new(&args.parquet_file, args.epochs)? // NEW: Specify run directory with metadata let run_id = format!("run_{}_hyperopt", chrono::Utc::now().format("%Y%m%d_%H%M%S")); let trainer = Mamba2Trainer::new(&args.parquet_file, args.epochs)? .with_checkpoint_dir(format!("/runpod-volume/training_runs/mamba2/{}", run_id)) .with_metadata(run_metadata); // New method to set metadata ``` ### Summary of Path Changes | Component | Old Path | New Path | |---|---|---| | Binaries | `/runpod-volume/binaries/train_dqn` | `/runpod-volume/binaries/current/train_dqn` (symlink to versioned) | | Datasets (Futures) | `/runpod-volume/test_data/ES_FUT_180d.parquet` | `/runpod-volume/datasets/parquet/futures/ES_FUT_180d.parquet` | | Datasets (Small) | `/runpod-volume/test_data/ES_FUT_small.parquet` | `/runpod-volume/datasets/parquet/futures_small/ES_FUT_small.parquet` | | Checkpoints | `/runpod-volume/checkpoints/mamba2_hyperopt/` | `/runpod-volume/training_runs/mamba2/run_YYYYMMDD_HHMMSS_hyperopt/checkpoints/` | | Metadata | N/A | `/runpod-volume/training_runs/mamba2/run_YYYYMMDD_HHMMSS_hyperopt/metadata.json` | | Hyperopt Logs | N/A | `/runpod-volume/training_runs/mamba2/run_YYYYMMDD_HHMMSS_hyperopt/logs/hyperopt.log` | | Production Models | N/A | `/runpod-volume/production_models/mamba2/v1_YYYYMMDD_HHMMSS/model.safetensors` | --- ## 6. S3 Utility Functions ### 6.1 Run ID Generation ```rust // ml/src/s3/run_id.rs use chrono::Utc; pub fn generate_run_id(model_type: &str, run_type: &str) -> String { format!( "run_{}_{}_{}", Utc::now().format("%Y%m%d_%H%M%S"), model_type, run_type ) } // Examples: // generate_run_id("mamba2", "hyperopt") → "run_20251028_224530_mamba2_hyperopt" // generate_run_id("dqn", "manual") → "run_20251028_153045_dqn_manual" ``` ### 6.2 Training Run Directory Structure Creation ```rust // ml/src/s3/run_setup.rs use std::fs; use std::path::{Path, PathBuf}; use anyhow::Result; pub struct TrainingRunSetup { pub run_id: String, pub model_type: String, pub base_dir: PathBuf, pub checkpoint_dir: PathBuf, pub hyperopt_dir: Option, pub logs_dir: PathBuf, pub metrics_dir: PathBuf, pub metadata_path: PathBuf, } impl TrainingRunSetup { pub fn new(model_type: &str, run_type: &str) -> Self { let run_id = generate_run_id(model_type, run_type); let base_dir = PathBuf::from(format!( "/runpod-volume/training_runs/{}/{}", model_type, run_id )); let checkpoint_dir = base_dir.join("checkpoints"); let hyperopt_dir = if run_type == "hyperopt" { Some(base_dir.join("hyperopt")) } else { None }; let logs_dir = base_dir.join("logs"); let metrics_dir = base_dir.join("metrics"); let metadata_path = base_dir.join("metadata.json"); Self { run_id, model_type: model_type.to_string(), base_dir, checkpoint_dir, hyperopt_dir, logs_dir, metrics_dir, metadata_path, } } pub fn create_directories(&self) -> Result<()> { fs::create_dir_all(&self.checkpoint_dir)?; if let Some(ref hyperopt_dir) = self.hyperopt_dir { fs::create_dir_all(hyperopt_dir)?; } fs::create_dir_all(&self.logs_dir)?; fs::create_dir_all(&self.metrics_dir)?; Ok(()) } } ``` ### 6.3 Metadata Saving ```rust // ml/src/s3/metadata.rs use serde::{Serialize, Deserialize}; use std::fs::File; use std::path::Path; use anyhow::Result; #[derive(Serialize, Deserialize)] pub struct TrainingMetadata { pub run_id: String, pub model_type: String, pub run_type: String, pub created_at: String, pub git_commit: String, pub git_branch: String, pub dataset: DatasetInfo, pub hardware: HardwareInfo, pub training_config: serde_json::Value, pub hyperparameters: serde_json::Value, } pub fn save_metadata>( metadata: &TrainingMetadata, path: P, ) -> Result<()> { let file = File::create(path)?; serde_json::to_writer_pretty(file, metadata)?; Ok(()) } pub fn load_metadata>(path: P) -> Result { let file = File::open(path)?; let metadata = serde_json::from_reader(file)?; Ok(metadata) } ``` ### 6.4 Hyperopt Trial Tracking ```rust // ml/src/s3/hyperopt_tracker.rs use serde::{Serialize, Deserialize}; use std::fs::File; use std::path::Path; use anyhow::Result; #[derive(Serialize, Deserialize)] pub struct HyperoptTrialHistory { pub study_name: String, pub created_at: String, pub total_trials: usize, pub best_trial_number: usize, pub best_value: f64, pub optimization_direction: String, pub trials: Vec, } #[derive(Serialize, Deserialize)] pub struct TrialInfo { pub trial_number: usize, pub state: String, pub value: f64, pub datetime_start: String, pub datetime_complete: String, pub duration_seconds: f64, pub params: serde_json::Value, pub user_attrs: serde_json::Value, } pub fn save_trial_history>( history: &HyperoptTrialHistory, path: P, ) -> Result<()> { let file = File::create(path)?; serde_json::to_writer_pretty(file, history)?; Ok(()) } pub fn append_trial>( trial: TrialInfo, path: P, ) -> Result<()> { let mut history: HyperoptTrialHistory = if path.as_ref().exists() { let file = File::open(&path)?; serde_json::from_reader(file)? } else { HyperoptTrialHistory { study_name: "".to_string(), created_at: chrono::Utc::now().to_rfc3339(), total_trials: 0, best_trial_number: 0, best_value: f64::INFINITY, optimization_direction: "minimize".to_string(), trials: Vec::new(), } }; history.trials.push(trial); history.total_trials = history.trials.len(); // Update best if improved if trial.value < history.best_value { history.best_value = trial.value; history.best_trial_number = trial.trial_number; } save_trial_history(&history, path) } ``` ### 6.5 S3 Path Resolution ```rust // ml/src/s3/paths.rs use std::path::PathBuf; pub const S3_BASE: &str = "/runpod-volume"; pub struct S3Paths; impl S3Paths { pub fn binary_current(name: &str) -> PathBuf { PathBuf::from(format!("{}/binaries/current/{}", S3_BASE, name)) } pub fn dataset_futures(symbol: &str) -> PathBuf { PathBuf::from(format!("{}/datasets/parquet/futures/{}", S3_BASE, symbol)) } pub fn dataset_futures_small(symbol: &str) -> PathBuf { PathBuf::from(format!("{}/datasets/parquet/futures_small/{}", S3_BASE, symbol)) } pub fn training_run_base(model_type: &str, run_id: &str) -> PathBuf { PathBuf::from(format!("{}/training_runs/{}/{}", S3_BASE, model_type, run_id)) } pub fn production_model(model_type: &str, version: &str) -> PathBuf { PathBuf::from(format!("{}/production_models/{}/{}", S3_BASE, model_type, version)) } pub fn production_model_latest(model_type: &str) -> PathBuf { PathBuf::from(format!("{}/production_models/{}/latest", S3_BASE, model_type)) } } // Usage: // let dataset = S3Paths::dataset_futures("ES_FUT_180d.parquet"); // let binary = S3Paths::binary_current("train_mamba2_parquet"); ``` --- ## 7. Implementation Timeline ### Phase 1: Structure Creation (Day 1, 1 hour) - Create new S3 directory structure - Write migration scripts - Test on small dataset ### Phase 2: Binary Migration (Day 1, 2 hours) - Migrate binaries to versioned structure - Create symlinks (metadata files) - Update build scripts to generate versioned binaries ### Phase 3: Dataset Migration (Day 1, 1 hour) - Reorganize datasets into structured hierarchy - Create dataset catalog ### Phase 4: Code Updates (Day 2-3, 8 hours) - Update hyperopt adapters - Update training examples - Update deployment scripts - Add S3 utility functions - Write tests ### Phase 5: Existing Run Migration (Day 3, 2 hours) - Migrate existing DQN run with metadata - Generate metadata from available info - Test retrieval ### Phase 6: Testing (Day 4, 4 hours) - Run full hyperopt workflow with new structure - Verify metadata generation - Test production promotion workflow - Validate backward compatibility ### Phase 7: Documentation (Day 4, 2 hours) - Update RUNPOD_VOLUME_MOUNT_ARCHITECTURE.md - Update ML_TRAINING_PARQUET_GUIDE.md - Create S3_ORGANIZATION_GUIDE.md ### Phase 8: Cleanup (Day 5, 2 hours) - Remove old flat structures (after validation) - Update all documentation references - Deploy to production **Total Estimated Time**: 22 hours over 5 days --- ## 8. Cost Analysis ### Storage Costs - **Volume**: $5/month (50GB, currently 103.9 MB = 0.2% used) - **Headroom**: 49.9 GB available (485x current usage) - **Per-Run Overhead**: ~5 MB (metadata + logs) - **Capacity**: ~10,000 training runs before hitting limit ### Training Costs (Unchanged) - **RTX A4000**: $0.25/hr (~$0.10 per 30-min hyperopt run) - **Tesla V100**: $0.10/hr (~$0.04 per 30-min hyperopt run) ### Migration Costs - **One-time**: ~1 hour of developer time + 2 hours of GPU time for testing - **Recurring**: None (structure maintenance is automated) --- ## 9. Benefits Summary ### Immediate Benefits 1. **Reproducibility**: Full training context captured (git commit, hyperparams, dataset) 2. **Hyperopt Tracking**: All trials logged with Optuna-compatible format 3. **Production Promotion**: Clear workflow for promoting models to production 4. **Binary Versioning**: No more "which version broke?" debugging 5. **Dataset Organization**: Easy to find and use correct datasets ### Long-Term Benefits 1. **Scalability**: Structure supports hundreds of training runs 2. **Auditability**: Complete audit trail for compliance 3. **Collaboration**: Team can easily share and understand runs 4. **Analysis**: Rich metadata enables cross-run analysis 5. **Rollback**: Easy to revert to previous production models ### Operational Benefits 1. **Zero Downtime**: Migration is additive, not destructive 2. **Backward Compatible**: Old paths remain until code is updated 3. **Self-Documenting**: Metadata is human-readable JSON 4. **Tool-Friendly**: Works with AWS CLI, Python boto3, and Rust aws-sdk-s3 --- ## 10. Next Steps 1. **Review**: Get approval from team on structure design 2. **Implement**: Execute migration plan (Phase 1-3, ~4 hours) 3. **Code Updates**: Update Rust codebase (Phase 4-5, ~10 hours) 4. **Test**: Run full hyperopt workflow on new structure (Phase 6, ~4 hours) 5. **Deploy**: Update deployment scripts and documentation (Phase 7-8, ~4 hours) 6. **Monitor**: Track first 10 training runs for issues **Target Go-Live**: 2025-11-01 (4 days from now) --- **Document Version**: 1.0 **Last Updated**: 2025-10-28 **Next Review**: After first production hyperopt run