Files

jgrusewski 9c3d741a08 refactor: restructure repo — crates/, bin/, testing/ layout

Move 17 library crates into crates/, CLI binary into bin/fxt,
consolidate 10 test crates into testing/, split config crate
from deployment config files.

Root directory reduced from 38+ to ~17 directories.
All Cargo.toml paths and build.rs proto refs updated.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

2026-02-25 11:56:00 +01:00

2.8 KiB

Raw Blame History

GPU Training Benchmark - Quick Start

Prerequisites

RTX 3050 Ti GPU (4GB VRAM) or equivalent
CUDA 12.8+ installed
360 DBN files in test_data/real/databento/ml_training/

1. Run Benchmark (Default: 10 epochs)

cd /home/jgrusewski/Work/foxhunt
cargo run -p ml --example gpu_training_benchmark --release

Expected Duration: 30-60 minutes Output: ml/benchmark_results/gpu_training_benchmark_{timestamp}.json

2. Interpret Results

The JSON report contains:

dqn_results.statistics.mean_seconds - Average epoch time for DQN
ppo_results.statistics.mean_seconds - Average epoch time for PPO
decision.recommendation - "local_gpu" or "cloud_gpu"
decision.rationale - Why this recommendation

3. Decision Framework

Recommendation: "local_gpu"

Total training time < 24 hours
Action: Proceed with local RTX 3050 Ti training
Cost: $0 (local hardware)

Recommendation: "cloud_gpu"

Total training time > 48 hours
Action: Use cloud GPU (A100 recommended)
Cost: ~$250/week

Recommendation: "either"

Total training time 24-48 hours
Action: User choice based on urgency/cost
Compare estimated_cost_local vs estimated_cost_cloud

4. Advanced Options

Custom epochs (more accurate)

cargo run -p ml --example gpu_training_benchmark --release -- --epochs 20

CPU-only mode (testing)

cargo run -p ml --example gpu_training_benchmark --release -- --cpu-only

Custom output path

cargo run -p ml --example gpu_training_benchmark --release -- --output my_results.json

5. Troubleshooting

Error: "CUDA out of memory"

TFT model requires 1.5-2.5GB VRAM
Solution: Batch size automatically limited to 4
If still OOM: Close other GPU applications

Warning: "Thermal throttling detected"

GPU temperature >85°C
Solution: Improve cooling, reduce room temperature
Training will continue but slower

Error: "DBN files not found"

Missing training data

Solution: Run download script first:

cargo run -p data --example download_ml_training_data --release

6. Full Documentation

See ml/docs/GPU_BENCHMARK_GUIDE.md for comprehensive documentation.

Expected Output Example

{
  "timestamp": "2025-10-13T12:30:00Z",
  "decision": {
    "recommendation": "local_gpu",
    "rationale": "Training time 18.5 hours < 24h threshold. Local GPU viable.",
    "estimated_local_hours": 18.5,
    "estimated_cost_local": 0.42,
    "estimated_cost_cloud": 9.73
  },
  "dqn_results": {
    "statistics": {
      "mean_seconds": 0.045,
      "confidence_interval_95": [0.042, 0.048],
      "p95": 0.052,
      "p99": 0.058
    },
    "memory_peak_mb": 127.3,
    "stability": {"is_stable": true}
  },
  "ppo_results": { ... }
}

Questions? See full documentation or troubleshooting guide.

2.8 KiB Raw Blame History