foxhunt/docs/archive/wave_d/reports/S3_ORGANIZATION_DESIGN.md

Production S3 Organization Structure for ML Training

Created: 2025-10-28 Bucket: s3://se3zdnb5o4 (Runpod EUR-IS-1) Status: Design Complete - Ready for Implementation

---

Executive Summary

This document defines a production-ready S3 organization structure for ML training on Runpod. The design scales to hundreds of training runs, supports hyperparameter optimization metadata, and enables efficient retrieval of models and results.

Key Features:

• Versioned binaries with timestamps
• Hierarchical training run organization
• Rich metadata capture (hyperparams, git commit, metrics)
• Hyperopt trial tracking with Optuna integration
• Production model promotion workflow
• Zero-downtime migration from current flat structure

---

1. Current S3 Structure Analysis

Current Organization (Flat, Unstructured)

s3://se3zdnb5o4/
├── .env                                    (1.5 KB)
├── AGENT_5_S3_UPLOAD_REPORT.md            (8.0 KB)
├── binaries/                               [80.2 MB, 5 files]
│   ├── CHECKSUMS.txt
│   ├── hyperopt_mamba2_demo               (21 MB, multiple versions)
│   ├── hyperopt_mamba2_demo_CUDA12        (duplicates)
│   ├── hyperopt_mamba2_demo_CUDA12_20251027_234229 (timestamped)
│   ├── hyperopt_mamba2_demo_FIXED         (more duplicates)
│   ├── train_dqn                          (17.8 MB)
│   ├── train_mamba2_dbn                   (13.9 MB)
│   ├── train_mamba2_parquet               (17.8 MB)
│   ├── train_mamba2_parquet_ADAMW_FIX     (20.7 MB)
│   ├── train_ppo                          (11.4 MB)
│   ├── train_ppo_parquet                  (20.7 MB)
│   ├── train_tft_parquet                  (18.5 MB)
│   └── (multiple checksum files)
├── checkpoints/                            [30 bytes]
│   └── README.txt
├── debug_tests/                            [7.3 MB]
│   ├── test1_hello
│   └── test2_cuda_check
├── logs/                                   [26 bytes]
│   └── README.txt
├── models/                                 [1.85 MB, 12 files] - FLAT!
│   ├── dqn_epoch_10.safetensors
│   ├── dqn_epoch_20.safetensors
│   ├── ... (epochs 30-90)
│   ├── dqn_epoch_100.safetensors
│   ├── dqn_final_epoch1.safetensors
│   ├── dqn_final_epoch100.safetensors
│   └── tft_180d_50ep_rtx4090_bs64/
│       ├── tft_225_epoch_0.json
│       └── tft_225_epoch_0.safetensors
└── test_data/                              [12.8 MB]
    ├── (9 parquet files - unorganized)
    └── real/
        ├── databento/ml_training/ (392 .dbn files)
        └── parquet/ (2 crypto files)

Issues Identified

1. Binary Versioning Chaos: 5+ versions of hyperopt_mamba2_demo with unclear naming
2. No Training Run Organization: All DQN checkpoints in flat /models/ directory
3. Missing Metadata: No run ID, hyperparameters, git commit, or trial info
4. No Hyperopt Tracking: Can't track Optuna trials or best parameters
5. Unclear Production Status: No way to identify "promoted" production models
6. Data Duplication: Multiple dataset locations (test_data/, test_data/real/)
7. No Logs: Empty /logs/ directory - training logs not captured
8. Checksum Files Everywhere: 4 different CHECKSUMS.txt files in /binaries/

---

2. Production S3 Directory Structure

Complete Hierarchy

s3://se3zdnb5o4/
├── binaries/                               # Versioned training binaries
│   ├── current/                            # Latest stable versions (symlinks)
│   │   ├── train_dqn -> ../v20251028_134530/train_dqn
│   │   ├── train_mamba2_parquet -> ../v20251027_225930/train_mamba2_parquet
│   │   ├── train_ppo -> ../v20251028_134530/train_ppo
│   │   ├── train_tft_parquet -> ../v20251028_134530/train_tft_parquet
│   │   └── hyperopt_mamba2_demo -> ../v20251028_224635/hyperopt_mamba2_demo
│   ├── v20251028_134530/                   # Version: YYYYMMDD_HHMMSS
│   │   ├── train_dqn                       (17.8 MB)
│   │   ├── train_ppo                       (11.4 MB)
│   │   ├── train_tft_parquet               (18.5 MB)
│   │   ├── checksums.sha256                # SHA256 hashes for this version
│   │   └── build_metadata.json             # Git commit, build time, Rust version
│   ├── v20251028_224635/
│   │   ├── hyperopt_mamba2_demo            (21 MB)
│   │   ├── checksums.sha256
│   │   └── build_metadata.json
│   └── v20251027_225930/
│       ├── train_mamba2_parquet            (17.8 MB)
│       ├── checksums.sha256
│       └── build_metadata.json
│
├── datasets/                               # Organized training data
│   ├── parquet/                            # Parquet format (preferred)
│   │   ├── futures/
│   │   │   ├── ES_FUT_180d.parquet         (2.9 MB)
│   │   │   ├── NQ_FUT_180d.parquet         (4.3 MB)
│   │   │   ├── 6E_FUT_180d.parquet         (2.7 MB)
│   │   │   └── ZN_FUT_90d.parquet          (2.7 MB)
│   │   ├── futures_small/                  # Smoke test datasets
│   │   │   ├── ES_FUT_small.parquet        (24.7 KB)
│   │   │   ├── NQ_FUT_small.parquet        (26.6 KB)
│   │   │   ├── 6E_FUT_small.parquet        (22.3 KB)
│   │   │   └── ZN_FUT_small.parquet        (18.8 KB)
│   │   └── crypto/
│   │       ├── BTC-USD_30day_2024-09.parquet (871 KB)
│   │       └── ETH-USD_30day_2024-09.parquet (800 KB)
│   └── dbn/                                # Databento format (historical)
│       └── ohlcv-1m/
│           ├── ES.FUT/
│           │   └── (98 .dbn files)
│           ├── NQ.FUT/
│           │   └── (98 .dbn files)
│           ├── 6E.FUT/
│           │   └── (98 .dbn files)
│           └── ZN.FUT/
│               └── (98 .dbn files)
│
├── training_runs/                          # All training runs (hyperopt + manual)
│   ├── mamba2/
│   │   ├── run_20251028_224530_hyperopt/   # Hyperopt run
│   │   │   ├── metadata.json               # Run config, git commit, dataset
│   │   │   ├── hyperopt/
│   │   │   │   ├── study.db                # Optuna SQLite database
│   │   │   │   ├── trial_history.json      # All trials (params + metrics)
│   │   │   │   ├── best_params.json        # Best hyperparameters found
│   │   │   │   └── optuna_visualization.html # Plots (optional)
│   │   │   ├── checkpoints/
│   │   │   │   ├── trial_0_epoch_5.safetensors
│   │   │   │   ├── trial_0_epoch_10.safetensors
│   │   │   │   ├── trial_1_epoch_5.safetensors
│   │   │   │   ├── ...
│   │   │   │   └── best_model.safetensors  # Best overall model
│   │   │   ├── logs/
│   │   │   │   ├── hyperopt.log            # Hyperopt progress
│   │   │   │   ├── trial_0.log             # Individual trial logs
│   │   │   │   ├── trial_1.log
│   │   │   │   └── ...
│   │   │   └── metrics/
│   │   │       ├── training_curves.png     # Loss/accuracy plots
│   │   │       ├── param_importance.png    # Hyperparameter sensitivity
│   │   │       └── convergence.png         # Optimization convergence
│   │   ├── run_20251028_153045_manual/     # Manual training run
│   │   │   ├── metadata.json
│   │   │   ├── checkpoints/
│   │   │   │   ├── epoch_5.safetensors
│   │   │   │   ├── epoch_10.safetensors
│   │   │   │   ├── ...
│   │   │   │   └── final_model.safetensors
│   │   │   ├── logs/
│   │   │   │   └── training.log
│   │   │   └── metrics/
│   │   │       ├── training_losses.csv
│   │   │       └── training_metrics.json
│   │   └── run_20251027_180000_baseline/   # Baseline comparison run
│   │       └── ...
│   ├── tft/
│   │   ├── run_20251028_100000_hyperopt/
│   │   │   └── ... (same structure)
│   │   └── run_20251028_120000_manual/
│   │       └── ...
│   ├── dqn/
│   │   ├── run_20251025_005300_manual/     # Existing DQN run (migrated)
│   │   │   ├── metadata.json
│   │   │   ├── checkpoints/
│   │   │   │   ├── epoch_10.safetensors
│   │   │   │   ├── epoch_20.safetensors
│   │   │   │   ├── ...
│   │   │   │   └── final_epoch_100.safetensors
│   │   │   ├── logs/
│   │   │   │   └── training.log (empty - not captured)
│   │   │   └── metrics/
│   │   │       └── (empty - not captured)
│   │   └── run_20251028_140000_hyperopt/
│   │       └── ...
│   └── ppo/
│       ├── run_20251028_160000_hyperopt/
│       │   └── ...
│       └── run_20251028_170000_manual/
│           └── ...
│
└── production_models/                      # Promoted production-ready models
    ├── mamba2/
    │   ├── v1_20251028_224530/             # Production version 1
    │   │   ├── model.safetensors           # Promoted from training_runs/
    │   │   ├── metadata.json               # Copy of training metadata
    │   │   ├── validation_report.json      # Production validation results
    │   │   └── promotion_notes.md          # Why this model was promoted
    │   └── latest -> v1_20251028_224530    # Symlink to latest version
    ├── tft/
    │   ├── v1_20251028_120000/
    │   │   └── ...
    │   └── latest -> v1_20251028_120000
    ├── dqn/
    │   ├── v1_20251025_005300/
    │   │   └── ...
    │   └── latest -> v1_20251025_005300
    └── ppo/
        ├── v1_20251028_170000/
        │   └── ...
        └── latest -> v1_20251028_170000

---

3. Metadata Schemas

3.1 Training Run Metadata (metadata.json)

Captures complete training context for reproducibility.

{
  "run_id": "run_20251028_224530_hyperopt",
  "model_type": "mamba2",
  "run_type": "hyperopt",
  "created_at": "2025-10-28T22:45:30Z",
  "git_commit": "3f4aae20",
  "git_branch": "main",
  "git_dirty": false,
  "binary_version": "v20251028_134530",
  "binary_checksum": "8063275fb2db1252f879b5d7852680b140b2c41b56403a7e496a3a3ebed2b692",
  "dataset": {
    "path": "s3://se3zdnb5o4/datasets/parquet/futures/ES_FUT_180d.parquet",
    "symbol": "ES.FUT",
    "timeframe": "1m",
    "bars": 259200,
    "date_range": ["2024-01-02", "2024-06-30"],
    "checksum": "sha256:..."
  },
  "hardware": {
    "gpu_type": "RTX A4000",
    "vram_gb": 16,
    "cuda_version": "12.9.1",
    "driver_version": "550.54.15"
  },
  "training_config": {
    "epochs": 50,
    "batch_size": 32,
    "learning_rate": 0.0001,
    "sequence_length": 60,
    "train_split": 0.8,
    "early_stopping_patience": 20
  },
  "hyperparameters": {
    "d_model": 225,
    "d_state": 16,
    "num_layers": 6,
    "dropout": 0.1,
    "weight_decay": 0.0001,
    "grad_clip": 1.0,
    "warmup_steps": 1000,
    "adam_beta1": 0.9,
    "adam_beta2": 0.999,
    "adam_epsilon": 1e-8,
    "lookback_window": 60,
    "sequence_stride": 1,
    "norm_eps": 1e-5
  },
  "training_duration": {
    "wall_time_seconds": 1860,
    "epochs_completed": 50,
    "best_epoch": 42
  },
  "final_metrics": {
    "train_loss": 0.003245,
    "val_loss": 0.003892,
    "perplexity": 1.0039,
    "directional_accuracy": 0.62,
    "mae": 0.0042,
    "rmse": 0.0068,
    "r_squared": 0.84
  },
  "tags": ["production_candidate", "rtx_a4000_optimized"]
}

3.2 Hyperopt Trial History (trial_history.json)

Optuna-compatible trial tracking for analysis.

{
  "study_name": "mamba2_hyperopt_20251028_224530",
  "created_at": "2025-10-28T22:45:30Z",
  "total_trials": 30,
  "best_trial_number": 18,
  "best_value": 0.003892,
  "optimization_direction": "minimize",
  "search_space": {
    "learning_rate": {"type": "log_uniform", "low": 1e-5, "high": 1e-2},
    "batch_size": {"type": "int", "low": 4, "high": 96},
    "dropout": {"type": "uniform", "low": 0.0, "high": 0.5},
    "weight_decay": {"type": "log_uniform", "low": 1e-6, "high": 1e-2},
    "grad_clip": {"type": "log_uniform", "low": 0.5, "high": 5.0},
    "warmup_steps": {"type": "int", "low": 100, "high": 2000},
    "adam_beta1": {"type": "uniform", "low": 0.85, "high": 0.95},
    "adam_beta2": {"type": "uniform", "low": 0.98, "high": 0.999},
    "adam_epsilon": {"type": "log_uniform", "low": 1e-9, "high": 1e-7},
    "lookback_window": {"type": "int", "low": 30, "high": 120},
    "sequence_stride": {"type": "int", "low": 1, "high": 5},
    "norm_eps": {"type": "log_uniform", "low": 1e-6, "high": 1e-4}
  },
  "trials": [
    {
      "trial_number": 0,
      "state": "COMPLETE",
      "value": 0.008234,
      "datetime_start": "2025-10-28T22:45:35Z",
      "datetime_complete": "2025-10-28T22:47:52Z",
      "duration_seconds": 137,
      "params": {
        "learning_rate": 0.000123,
        "batch_size": 32,
        "dropout": 0.15,
        "weight_decay": 0.00015,
        "grad_clip": 1.2,
        "warmup_steps": 500,
        "adam_beta1": 0.9,
        "adam_beta2": 0.999,
        "adam_epsilon": 1e-8,
        "lookback_window": 60,
        "sequence_stride": 1,
        "norm_eps": 1e-5
      },
      "user_attrs": {
        "train_loss": 0.007891,
        "epochs_completed": 50,
        "gpu_memory_peak_mb": 1420
      }
    },
    {
      "trial_number": 1,
      "state": "COMPLETE",
      "value": 0.006123,
      "datetime_start": "2025-10-28T22:47:55Z",
      "datetime_complete": "2025-10-28T22:50:10Z",
      "duration_seconds": 135,
      "params": { "...": "..." },
      "user_attrs": { "...": "..." }
    }
  ]
}

3.3 Best Hyperparameters (best_params.json)

Quick reference for production deployment.

{
  "study_name": "mamba2_hyperopt_20251028_224530",
  "best_trial_number": 18,
  "best_value": 0.003892,
  "found_at": "2025-10-28T23:32:15Z",
  "params": {
    "learning_rate": 0.000087,
    "batch_size": 48,
    "dropout": 0.12,
    "weight_decay": 0.00012,
    "grad_clip": 1.5,
    "warmup_steps": 800,
    "adam_beta1": 0.91,
    "adam_beta2": 0.9995,
    "adam_epsilon": 2.3e-8,
    "lookback_window": 75,
    "sequence_stride": 2,
    "norm_eps": 3.5e-6
  },
  "metrics": {
    "train_loss": 0.003421,
    "val_loss": 0.003892,
    "directional_accuracy": 0.64,
    "mae": 0.0038
  },
  "checkpoint_path": "s3://se3zdnb5o4/training_runs/mamba2/run_20251028_224530_hyperopt/checkpoints/best_model.safetensors",
  "recommended_for_production": true,
  "notes": "Best model found after 18 trials. Significantly better directional accuracy (64% vs 58% baseline)."
}

3.4 Binary Build Metadata (build_metadata.json)

Tracks binary provenance for debugging.

{
  "version": "v20251028_134530",
  "build_timestamp": "2025-10-28T13:45:30Z",
  "git_commit": "3f4aae20",
  "git_branch": "main",
  "git_dirty": false,
  "git_remote": "https://github.com/foxhunt/ml.git",
  "rust_version": "1.75.0",
  "cargo_profile": "release",
  "features": ["cuda", "mimalloc-allocator"],
  "target_triple": "x86_64-unknown-linux-gnu",
  "cuda_version": "12.9.1",
  "binaries": {
    "train_dqn": {
      "size_bytes": 17875296,
      "sha256": "8412e3426ca7d53e2db18a0181656649f7398aff392d0889ed18c6d9e488a93f"
    },
    "train_ppo": {
      "size_bytes": 11440200,
      "sha256": "e5b6b566c85ec83cd118332c986a78ca1fa1580781b5515b84b80a391f5c32da"
    },
    "train_tft_parquet": {
      "size_bytes": 18578960,
      "sha256": "47061c765ae8568da238d52dd93993ed9f4b0cfaf003206699a01047cbd22d52"
    }
  },
  "build_host": {
    "hostname": "build-server-1",
    "os": "Linux 6.14.0-33-generic",
    "arch": "x86_64"
  }
}

3.5 Production Promotion Metadata (validation_report.json)

Justifies production deployment.

{
  "model_id": "mamba2_v1_20251028_224530",
  "promoted_from": "s3://se3zdnb5o4/training_runs/mamba2/run_20251028_224530_hyperopt/checkpoints/best_model.safetensors",
  "promoted_at": "2025-10-29T10:15:00Z",
  "promoted_by": "trading_team",
  "validation_results": {
    "holdout_dataset": "ES_FUT_2024-07-01_to_2024-09-30",
    "holdout_metrics": {
      "val_loss": 0.004012,
      "directional_accuracy": 0.63,
      "sharpe_ratio": 2.15,
      "max_drawdown": 0.12,
      "win_rate": 0.61
    },
    "backtesting": {
      "period": "2024-07-01 to 2024-09-30",
      "initial_capital": 100000,
      "final_equity": 128450,
      "total_return": 0.2845,
      "sharpe_ratio": 2.15,
      "max_drawdown": 0.12,
      "win_rate": 0.61,
      "profit_factor": 1.85
    }
  },
  "comparison_to_baseline": {
    "baseline_model": "mamba2_default_params",
    "improvement_directional_accuracy": "+8.5%",
    "improvement_sharpe": "+0.35",
    "improvement_drawdown": "-3.2%"
  },
  "production_notes": "Significantly improved directional accuracy over baseline. Ready for 10% live capital allocation.",
  "rollback_plan": "Revert to mamba2_v0_baseline if Sharpe < 1.5 over 7 days",
  "approval": {
    "approved_by": "risk_committee",
    "approval_date": "2025-10-29T09:00:00Z",
    "signatures": ["john_doe", "jane_smith"]
  }
}

---

4. Migration Plan

Phase 1: Create New Structure (Zero Downtime)

Duration: 5 minutes Risk: Low (no deletions, only additions)

# Create new directory structure
aws s3api put-object --bucket se3zdnb5o4 \
  --key binaries/current/.keep \
  --profile runpod --endpoint-url https://s3api-eur-is-1.runpod.io

aws s3api put-object --bucket se3zdnb5o4 \
  --key datasets/parquet/futures/.keep \
  --profile runpod --endpoint-url https://s3api-eur-is-1.runpod.io

aws s3api put-object --bucket se3zdnb5o4 \
  --key training_runs/mamba2/.keep \
  --profile runpod --endpoint-url https://s3api-eur-is-1.runpod.io

aws s3api put-object --bucket se3zdnb5o4 \
  --key production_models/mamba2/.keep \
  --profile runpod --endpoint-url https://s3api-eur-is-1.runpod.io

# (Repeat for all models: tft, dqn, ppo)

Phase 2: Migrate Binaries (Keep Old Paths)

Duration: 10 minutes Risk: Low (copies, not moves)

# Copy current binaries to versioned structure
VERSION="v20251028_$(date +%H%M%S)"

aws s3 cp s3://se3zdnb5o4/binaries/train_dqn \
  s3://se3zdnb5o4/binaries/$VERSION/train_dqn \
  --profile runpod --endpoint-url https://s3api-eur-is-1.runpod.io

# Create checksums
sha256sum train_dqn > checksums.sha256
aws s3 cp checksums.sha256 \
  s3://se3zdnb5o4/binaries/$VERSION/checksums.sha256 \
  --profile runpod --endpoint-url https://s3api-eur-is-1.runpod.io

# Create symlink metadata (S3 doesn't support symlinks, use JSON pointer)
echo '{"target": "'$VERSION'/train_dqn"}' > current_train_dqn.json
aws s3 cp current_train_dqn.json \
  s3://se3zdnb5o4/binaries/current/train_dqn.json \
  --profile runpod --endpoint-url https://s3api-eur-is-1.runpod.io

# Repeat for all binaries

Phase 3: Migrate Datasets (Reorganize)

Duration: 5 minutes Risk: Low (copies)

# Move parquet files to organized structure
aws s3 cp s3://se3zdnb5o4/test_data/ES_FUT_180d.parquet \
  s3://se3zdnb5o4/datasets/parquet/futures/ES_FUT_180d.parquet \
  --profile runpod --endpoint-url https://s3api-eur-is-1.runpod.io

aws s3 cp s3://se3zdnb5o4/test_data/ES_FUT_small.parquet \
  s3://se3zdnb5o4/datasets/parquet/futures_small/ES_FUT_small.parquet \
  --profile runpod --endpoint-url https://s3api-eur-is-1.runpod.io

# Repeat for all datasets (NQ, 6E, ZN, crypto)

# Keep old paths as copies for backward compatibility
# (Don't delete /test_data/ until all code is updated)

Phase 4: Migrate Existing Training Run (DQN)

Duration: 5 minutes Risk: Low (copies)

# Create metadata for existing DQN run
RUN_DIR="s3://se3zdnb5o4/training_runs/dqn/run_20251025_005300_manual"

# Generate metadata.json (from pod logs if available)
cat > metadata.json << 'EOF'
{
  "run_id": "run_20251025_005300_manual",
  "model_type": "dqn",
  "run_type": "manual",
  "created_at": "2025-10-25T00:53:00Z",
  "git_commit": "unknown",
  "dataset": {
    "path": "s3://se3zdnb5o4/test_data/ES_FUT_180d.parquet",
    "symbol": "ES.FUT"
  },
  "training_config": {
    "epochs": 100
  },
  "notes": "Migrated from flat /models/ structure. Original training logs not captured."
}
EOF

aws s3 cp metadata.json $RUN_DIR/metadata.json \
  --profile runpod --endpoint-url https://s3api-eur-is-1.runpod.io

# Copy checkpoints
aws s3 cp s3://se3zdnb5o4/models/dqn_epoch_10.safetensors \
  $RUN_DIR/checkpoints/epoch_10.safetensors \
  --profile runpod --endpoint-url https://s3api-eur-is-1.runpod.io

# Repeat for all epochs (10, 20, ..., 100)

# Copy final model
aws s3 cp s3://se3zdnb5o4/models/dqn_final_epoch100.safetensors \
  $RUN_DIR/checkpoints/final_epoch_100.safetensors \
  --profile runpod --endpoint-url https://s3api-eur-is-1.runpod.io

# Keep old paths for now (backward compatibility)

Phase 5: Update Code to Use New Paths

Duration: 2 hours Risk: Medium (requires testing)

See Section 5: Code Changes below.

Phase 6: Cleanup Old Structure (After Validation)

Duration: 10 minutes Risk: Medium (irreversible deletions)

# ONLY after confirming new structure works!
# Delete old flat structures
aws s3 rm s3://se3zdnb5o4/models/ --recursive \
  --profile runpod --endpoint-url https://s3api-eur-is-1.runpod.io

aws s3 rm s3://se3zdnb5o4/binaries/train_dqn \
  --profile runpod --endpoint-url https://s3api-eur-is-1.runpod.io

# (Keep old /test_data/ for backward compatibility until all code updated)

---

5. Code Changes Required

Files Requiring Path Updates

5.1 Hyperopt Adapters

Files:

• /home/jgrusewski/Work/foxhunt/ml/src/hyperopt/adapters/mamba2.rs
• /home/jgrusewski/Work/foxhunt/ml/src/hyperopt/adapters/tft.rs
• /home/jgrusewski/Work/foxhunt/ml/src/hyperopt/adapters/dqn.rs
• /home/jgrusewski/Work/foxhunt/ml/src/hyperopt/adapters/ppo.rs

Changes:

// OLD: Flat checkpoint directory
self.checkpoint_dir = PathBuf::from("/runpod-volume/checkpoints/mamba2_hyperopt");

// NEW: Timestamped run directory with hyperopt subdirectory
let run_id = format!("run_{}_hyperopt", chrono::Utc::now().format("%Y%m%d_%H%M%S"));
self.checkpoint_dir = PathBuf::from(format!("/runpod-volume/training_runs/mamba2/{}/checkpoints", run_id));

// Also save metadata
self.metadata_path = PathBuf::from(format!("/runpod-volume/training_runs/mamba2/{}/metadata.json", run_id));
self.hyperopt_dir = PathBuf::from(format!("/runpod-volume/training_runs/mamba2/{}/hyperopt", run_id));
self.logs_dir = PathBuf::from(format!("/runpod-volume/training_runs/mamba2/{}/logs", run_id));

5.2 Training Examples

Files:

• /home/jgrusewski/Work/foxhunt/ml/examples/train_mamba2_parquet.rs
• /home/jgrusewski/Work/foxhunt/ml/examples/train_tft_parquet.rs
• /home/jgrusewski/Work/foxhunt/ml/examples/train_dqn.rs
• /home/jgrusewski/Work/foxhunt/ml/examples/train_ppo.rs

Changes:

// OLD: Flat checkpoint directory
checkpoint_dir: PathBuf::from("ml/checkpoints/mamba2_parquet"),

// NEW: Timestamped run directory
let run_id = format!("run_{}_manual", chrono::Utc::now().format("%Y%m%d_%H%M%S"));
checkpoint_dir: PathBuf::from(format!("/runpod-volume/training_runs/mamba2/{}", run_id)),

// OLD: Hardcoded dataset path
parquet_file: PathBuf::from("test_data/ES_FUT_180d.parquet"),

// NEW: Organized dataset path
parquet_file: PathBuf::from("/runpod-volume/datasets/parquet/futures/ES_FUT_180d.parquet"),

5.3 Runpod Deployment Script

File: /home/jgrusewski/Work/foxhunt/scripts/runpod_deploy.py

Changes:

# OLD: Flat binary path
'--command', default='/runpod-volume/binaries/train_dqn ...'

# NEW: Current symlink (points to latest version)
'--command', default='/runpod-volume/binaries/current/train_dqn ...'

# NEW: Use organized dataset paths
--parquet-file /runpod-volume/datasets/parquet/futures/ES_FUT_180d.parquet

# NEW: Output to timestamped run directory
--output-dir /runpod-volume/training_runs/dqn/run_$(date +%Y%m%d_%H%M%S)_manual

5.4 Hyperopt Examples

Files:

• /home/jgrusewski/Work/foxhunt/ml/examples/hyperopt_mamba2_demo.rs
• /home/jgrusewski/Work/foxhunt/ml/examples/hyperopt_tft_demo.rs
• /home/jgrusewski/Work/foxhunt/ml/examples/hyperopt_dqn_demo.rs
• /home/jgrusewski/Work/foxhunt/ml/examples/hyperopt_ppo_demo.rs

Changes:

// OLD: Uses default checkpoint_dir from adapter
let trainer = Mamba2Trainer::new(&args.parquet_file, args.epochs)?

// NEW: Specify run directory with metadata
let run_id = format!("run_{}_hyperopt", chrono::Utc::now().format("%Y%m%d_%H%M%S"));
let trainer = Mamba2Trainer::new(&args.parquet_file, args.epochs)?
    .with_checkpoint_dir(format!("/runpod-volume/training_runs/mamba2/{}", run_id))
    .with_metadata(run_metadata);  // New method to set metadata

Summary of Path Changes

Component	Old Path	New Path
Binaries	/runpod-volume/binaries/train_dqn	/runpod-volume/binaries/current/train_dqn (symlink to versioned)
Datasets (Futures)	/runpod-volume/test_data/ES_FUT_180d.parquet	/runpod-volume/datasets/parquet/futures/ES_FUT_180d.parquet
Datasets (Small)	/runpod-volume/test_data/ES_FUT_small.parquet	/runpod-volume/datasets/parquet/futures_small/ES_FUT_small.parquet
Checkpoints	/runpod-volume/checkpoints/mamba2_hyperopt/	/runpod-volume/training_runs/mamba2/run_YYYYMMDD_HHMMSS_hyperopt/checkpoints/
Metadata	N/A	/runpod-volume/training_runs/mamba2/run_YYYYMMDD_HHMMSS_hyperopt/metadata.json
Hyperopt Logs	N/A	/runpod-volume/training_runs/mamba2/run_YYYYMMDD_HHMMSS_hyperopt/logs/hyperopt.log
Production Models	N/A	/runpod-volume/production_models/mamba2/v1_YYYYMMDD_HHMMSS/model.safetensors

---

6. S3 Utility Functions

6.1 Run ID Generation

// ml/src/s3/run_id.rs
use chrono::Utc;

pub fn generate_run_id(model_type: &str, run_type: &str) -> String {
    format!(
        "run_{}_{}_{}",
        Utc::now().format("%Y%m%d_%H%M%S"),
        model_type,
        run_type
    )
}

// Examples:
// generate_run_id("mamba2", "hyperopt") → "run_20251028_224530_mamba2_hyperopt"
// generate_run_id("dqn", "manual") → "run_20251028_153045_dqn_manual"

6.2 Training Run Directory Structure Creation

// ml/src/s3/run_setup.rs
use std::fs;
use std::path::{Path, PathBuf};
use anyhow::Result;

pub struct TrainingRunSetup {
    pub run_id: String,
    pub model_type: String,
    pub base_dir: PathBuf,
    pub checkpoint_dir: PathBuf,
    pub hyperopt_dir: Option<PathBuf>,
    pub logs_dir: PathBuf,
    pub metrics_dir: PathBuf,
    pub metadata_path: PathBuf,
}

impl TrainingRunSetup {
    pub fn new(model_type: &str, run_type: &str) -> Self {
        let run_id = generate_run_id(model_type, run_type);
        let base_dir = PathBuf::from(format!(
            "/runpod-volume/training_runs/{}/{}",
            model_type, run_id
        ));

        let checkpoint_dir = base_dir.join("checkpoints");
        let hyperopt_dir = if run_type == "hyperopt" {
            Some(base_dir.join("hyperopt"))
        } else {
            None
        };
        let logs_dir = base_dir.join("logs");
        let metrics_dir = base_dir.join("metrics");
        let metadata_path = base_dir.join("metadata.json");

        Self {
            run_id,
            model_type: model_type.to_string(),
            base_dir,
            checkpoint_dir,
            hyperopt_dir,
            logs_dir,
            metrics_dir,
            metadata_path,
        }
    }

    pub fn create_directories(&self) -> Result<()> {
        fs::create_dir_all(&self.checkpoint_dir)?;
        if let Some(ref hyperopt_dir) = self.hyperopt_dir {
            fs::create_dir_all(hyperopt_dir)?;
        }
        fs::create_dir_all(&self.logs_dir)?;
        fs::create_dir_all(&self.metrics_dir)?;
        Ok(())
    }
}

6.3 Metadata Saving

// ml/src/s3/metadata.rs
use serde::{Serialize, Deserialize};
use std::fs::File;
use std::path::Path;
use anyhow::Result;

#[derive(Serialize, Deserialize)]
pub struct TrainingMetadata {
    pub run_id: String,
    pub model_type: String,
    pub run_type: String,
    pub created_at: String,
    pub git_commit: String,
    pub git_branch: String,
    pub dataset: DatasetInfo,
    pub hardware: HardwareInfo,
    pub training_config: serde_json::Value,
    pub hyperparameters: serde_json::Value,
}

pub fn save_metadata<P: AsRef<Path>>(
    metadata: &TrainingMetadata,
    path: P,
) -> Result<()> {
    let file = File::create(path)?;
    serde_json::to_writer_pretty(file, metadata)?;
    Ok(())
}

pub fn load_metadata<P: AsRef<Path>>(path: P) -> Result<TrainingMetadata> {
    let file = File::open(path)?;
    let metadata = serde_json::from_reader(file)?;
    Ok(metadata)
}

6.4 Hyperopt Trial Tracking

// ml/src/s3/hyperopt_tracker.rs
use serde::{Serialize, Deserialize};
use std::fs::File;
use std::path::Path;
use anyhow::Result;

#[derive(Serialize, Deserialize)]
pub struct HyperoptTrialHistory {
    pub study_name: String,
    pub created_at: String,
    pub total_trials: usize,
    pub best_trial_number: usize,
    pub best_value: f64,
    pub optimization_direction: String,
    pub trials: Vec<TrialInfo>,
}

#[derive(Serialize, Deserialize)]
pub struct TrialInfo {
    pub trial_number: usize,
    pub state: String,
    pub value: f64,
    pub datetime_start: String,
    pub datetime_complete: String,
    pub duration_seconds: f64,
    pub params: serde_json::Value,
    pub user_attrs: serde_json::Value,
}

pub fn save_trial_history<P: AsRef<Path>>(
    history: &HyperoptTrialHistory,
    path: P,
) -> Result<()> {
    let file = File::create(path)?;
    serde_json::to_writer_pretty(file, history)?;
    Ok(())
}

pub fn append_trial<P: AsRef<Path>>(
    trial: TrialInfo,
    path: P,
) -> Result<()> {
    let mut history: HyperoptTrialHistory = if path.as_ref().exists() {
        let file = File::open(&path)?;
        serde_json::from_reader(file)?
    } else {
        HyperoptTrialHistory {
            study_name: "".to_string(),
            created_at: chrono::Utc::now().to_rfc3339(),
            total_trials: 0,
            best_trial_number: 0,
            best_value: f64::INFINITY,
            optimization_direction: "minimize".to_string(),
            trials: Vec::new(),
        }
    };

    history.trials.push(trial);
    history.total_trials = history.trials.len();

    // Update best if improved
    if trial.value < history.best_value {
        history.best_value = trial.value;
        history.best_trial_number = trial.trial_number;
    }

    save_trial_history(&history, path)
}

6.5 S3 Path Resolution

// ml/src/s3/paths.rs
use std::path::PathBuf;

pub const S3_BASE: &str = "/runpod-volume";

pub struct S3Paths;

impl S3Paths {
    pub fn binary_current(name: &str) -> PathBuf {
        PathBuf::from(format!("{}/binaries/current/{}", S3_BASE, name))
    }

    pub fn dataset_futures(symbol: &str) -> PathBuf {
        PathBuf::from(format!("{}/datasets/parquet/futures/{}", S3_BASE, symbol))
    }

    pub fn dataset_futures_small(symbol: &str) -> PathBuf {
        PathBuf::from(format!("{}/datasets/parquet/futures_small/{}", S3_BASE, symbol))
    }

    pub fn training_run_base(model_type: &str, run_id: &str) -> PathBuf {
        PathBuf::from(format!("{}/training_runs/{}/{}", S3_BASE, model_type, run_id))
    }

    pub fn production_model(model_type: &str, version: &str) -> PathBuf {
        PathBuf::from(format!("{}/production_models/{}/{}", S3_BASE, model_type, version))
    }

    pub fn production_model_latest(model_type: &str) -> PathBuf {
        PathBuf::from(format!("{}/production_models/{}/latest", S3_BASE, model_type))
    }
}

// Usage:
// let dataset = S3Paths::dataset_futures("ES_FUT_180d.parquet");
// let binary = S3Paths::binary_current("train_mamba2_parquet");

---

7. Implementation Timeline

Phase 1: Structure Creation (Day 1, 1 hour)

• Create new S3 directory structure
• Write migration scripts
• Test on small dataset

Phase 2: Binary Migration (Day 1, 2 hours)

• Migrate binaries to versioned structure
• Create symlinks (metadata files)
• Update build scripts to generate versioned binaries

Phase 3: Dataset Migration (Day 1, 1 hour)

• Reorganize datasets into structured hierarchy
• Create dataset catalog

Phase 4: Code Updates (Day 2-3, 8 hours)

• Update hyperopt adapters
• Update training examples
• Update deployment scripts
• Add S3 utility functions
• Write tests

Phase 5: Existing Run Migration (Day 3, 2 hours)

• Migrate existing DQN run with metadata
• Generate metadata from available info
• Test retrieval

Phase 6: Testing (Day 4, 4 hours)

• Run full hyperopt workflow with new structure
• Verify metadata generation
• Test production promotion workflow
• Validate backward compatibility

Phase 7: Documentation (Day 4, 2 hours)

• Update RUNPOD_VOLUME_MOUNT_ARCHITECTURE.md
• Update ML_TRAINING_PARQUET_GUIDE.md
• Create S3_ORGANIZATION_GUIDE.md

Phase 8: Cleanup (Day 5, 2 hours)

• Remove old flat structures (after validation)
• Update all documentation references
• Deploy to production

Total Estimated Time: 22 hours over 5 days

---

8. Cost Analysis

Storage Costs

• Volume: $5/month (50GB, currently 103.9 MB = 0.2% used)
• Headroom: 49.9 GB available (485x current usage)
• Per-Run Overhead: ~5 MB (metadata + logs)
• Capacity: ~10,000 training runs before hitting limit

Training Costs (Unchanged)

• RTX A4000: $0.25/hr (~$0.10 per 30-min hyperopt run)
• Tesla V100: $0.10/hr (~$0.04 per 30-min hyperopt run)

Migration Costs

• One-time: ~1 hour of developer time + 2 hours of GPU time for testing
• Recurring: None (structure maintenance is automated)

---

9. Benefits Summary

Immediate Benefits

1. Reproducibility: Full training context captured (git commit, hyperparams, dataset)
2. Hyperopt Tracking: All trials logged with Optuna-compatible format
3. Production Promotion: Clear workflow for promoting models to production
4. Binary Versioning: No more "which version broke?" debugging
5. Dataset Organization: Easy to find and use correct datasets

Long-Term Benefits

1. Scalability: Structure supports hundreds of training runs
2. Auditability: Complete audit trail for compliance
3. Collaboration: Team can easily share and understand runs
4. Analysis: Rich metadata enables cross-run analysis
5. Rollback: Easy to revert to previous production models

Operational Benefits

1. Zero Downtime: Migration is additive, not destructive
2. Backward Compatible: Old paths remain until code is updated
3. Self-Documenting: Metadata is human-readable JSON
4. Tool-Friendly: Works with AWS CLI, Python boto3, and Rust aws-sdk-s3

---

10. Next Steps

1. Review: Get approval from team on structure design
2. Implement: Execute migration plan (Phase 1-3, ~4 hours)
3. Code Updates: Update Rust codebase (Phase 4-5, ~10 hours)
4. Test: Run full hyperopt workflow on new structure (Phase 6, ~4 hours)
5. Deploy: Update deployment scripts and documentation (Phase 7-8, ~4 hours)
6. Monitor: Track first 10 training runs for issues

Target Go-Live: 2025-11-01 (4 days from now)

---

Document Version: 1.0 Last Updated: 2025-10-28 Next Review: After first production hyperopt run