foxhunt/docs/archive/wave_d/reports/HYPEROPT_DEPLOYMENT_VALIDATION.md

Hyperopt Deployment Validation - 2025-10-28 09:49 UTC

Pod Details

• Pod ID: qlql87w5avv1q1
• GPU: RTX A4000 16GB (requested, actual GPU TBD)
• Deployment: 2025-10-28 09:49:40 UTC
• Datacenter: EUR-IS-1
• Cost: $0.25/hr (actual, vs $0.17/hr estimated)
• Status: RUNNING (provisioning in progress)
• SSH: root@157.157.221.29:19735 (or qlql87w5avv1q1.ssh.runpod.io when ready)
• Jupyter: https://qlql87w5avv1q1-8888.proxy.runpod.net

Fixes Applied

1. Feature Normalization Fix (CRITICAL)

File: /home/jgrusewski/Work/foxhunt/ml/src/hyperopt/adapters/mamba2.rs (line 475-505)

Problem:

• Model received RAW prices ($5000-6000) as features
• Model targets were NORMALIZED [0, 1]
• Result: MSE = 25 million (catastrophic loss)

Solution:

// Compute feature normalization parameters ONCE from ALL features
let all_feature_values: Vec<f64> = features.iter()
    .flat_map(|f| f.iter().copied())
    .collect();

let feature_min = all_feature_values.iter()
    .copied()
    .fold(f64::INFINITY, f64::min);
let feature_max = all_feature_values.iter()
    .copied()
    .fold(f64::NEG_INFINITY, f64::max);

// NORMALIZE features to [0, 1] range
let sequence: Vec<f64> = features[window_idx..window_idx + seq_len]
    .iter()
    .flat_map(|f| f.iter().copied())
    .map(|val| (val - feature_min) / (feature_max - feature_min))  // ← NORMALIZE
    .collect();

Expected Impact:

• Train Loss Epoch 1: 0.08-0.20 (vs 408M before)
• Val Loss Epoch 1: 0.10-0.25 (vs similar catastrophic values)
• R² Epoch 1: 0.2-0.7 (vs -infinity before)
• Dir Acc Epoch 1: 58-65% (vs random 50% before)

2. CLI Batch Size Bounds

Already Implemented: --batch-size-max 144 CLI parameter

Benefits:

• Runtime GPU-specific tuning without recompilation
• Targets RTX A4000 16GB VRAM for optimal utilization
• Expected: 1.5× speedup (10 min/epoch vs 16 min before)

Deployment Command

python3 scripts/runpod_deploy.py \
  --gpu-type "RTX A4000" \
  --command "/runpod-volume/binaries/hyperopt_mamba2_demo \
    --parquet-file /runpod-volume/test_data/ES_FUT_180d.parquet \
    --trials 30 \
    --epochs 50 \
    --batch-size-max 144 \
    --n-initial 3"

Binary Details

• Path: /runpod-volume/binaries/hyperopt_mamba2_demo
• Size: 17.3 MiB (stripped from 21 MB)
• Built: 2025-10-28 10:48 UTC
• CUDA: 12.9.1 + cuDNN 9
• Uploaded: 2025-10-28 10:49 UTC to s3://se3zdnb5o4/binaries/

Monitoring Instructions

Wait for Pod Initialization (3-5 min)

# Check pod status
python3 -c "
import requests, os, json
from dotenv import load_dotenv
load_dotenv('.env.runpod')
api_key = os.getenv('RUNPOD_API_KEY')
response = requests.get(
    'https://rest.runpod.io/v1/pods/qlql87w5avv1q1',
    headers={'Authorization': f'Bearer {api_key}'}
)
print(json.dumps(response.json(), indent=2))
"

SSH Access (Once Pod is Ready)

# Option 1: Direct IP (when available)
ssh root@157.157.221.29 -p 19735

# Option 2: Runpod endpoint (when DNS propagates)
ssh root@qlql87w5avv1q1.ssh.runpod.io

# Check training logs
tail -f /workspace/logs/hyperopt_*.log
# or
journalctl -u training -f
# or
ps aux | grep hyperopt

Success Criteria for First Trial

1. Configuration Loads

Expected log output:

INFO Configuration:
INFO   Trials: 30
INFO   Epochs per trial: 50
INFO   Batch size bounds: [4, 144]
INFO   n_initial: 3
INFO Configuring batch_size bounds: [4, 144]

2. Feature Normalization Applied (CRITICAL)

Expected log output:

INFO Target normalization: min=5356.75, max=6811.75, range=1455.00
INFO Feature normalization: min=<value>, max=<value>, range=<value>

The feature normalization log is NEW and confirms the fix is working.

3. Losses in Valid Range (CRITICAL SUCCESS METRIC)

Expected log output:

INFO Trial 1/30: batch_size=<value>
INFO Epoch 1/50: Train Loss = 0.08-0.20, Val Loss = 0.10-0.25
INFO   Dir Acc = 58-65%
INFO   R² = 0.2-0.7

If losses > 1.0, FIX HAS FAILED - STOP IMMEDIATELY

4. GPU Utilization (Optimal Performance)

# SSH into pod and run:
nvidia-smi

# Expected:
# VRAM: 13-14GB / 16GB (81-88% utilization)
# GPU: 85-92% utilization
# Temp: 60-80°C

5. Trial Timing (Speedup Verification)

Expected log output:

INFO Epoch 1/50: Time = 10-11 min
INFO Epoch 2/50: Time = 10-11 min
...
INFO Trial 1/30: Total Time = 8.3-9.2 hours

Expected speedup: 1.5× (10 min/epoch vs 16 min before)

Validation Results

First Trial Metrics (TO BE FILLED AFTER 15-30 MIN)

Metric	Expected	Actual	Status
Train Loss Epoch 1	0.08-0.20	TBD	⏳
Val Loss Epoch 1	0.10-0.25	TBD	⏳
R² Epoch 1	0.2-0.7	TBD	⏳
Dir Acc Epoch 1	58-65%	TBD	⏳
VRAM Usage	13-14GB	TBD	⏳
GPU Util	> 85%	TBD	⏳
Epoch Time	~10 min	TBD	⏳

Status Legend:

• ⏳ Waiting for data
• ✅ Pass (within expected range)
• ⚠️ Warning (outside expected range but acceptable)
• ❌ Fail (critical issue, requires investigation)

Expected Final Results (After 30 trials, ~5.3 hours)

Metric	Estimate	Basis
Total Runtime	~5.3 hours	30 trials × 10.6 min/epoch × 50 epochs ÷ 60
Total Cost	$1.33	5.3 hrs × $0.25/hr
Best Val Loss	0.10-0.15	Based on TFT baseline
Best Dir Acc	62-68%	Based on TFT baseline
Best R²	0.5-0.8	Based on TFT baseline

Cost Comparison:

• Before optimization: $2.00 (8 hrs × $0.25/hr)
• After optimization: $1.33 (5.3 hrs × $0.25/hr)
• Savings: 33% ($0.67)

Troubleshooting

If Losses are Still > 1.0

1. SSH into pod
2. Check feature normalization log line exists:

   grep "Feature normalization" /workspace/logs/hyperopt_*.log
   

3. If missing, binary may not have new code
4. Verify binary upload timestamp:

   aws s3 ls s3://se3zdnb5o4/binaries/ --profile runpod \
     --endpoint-url https://s3api-eur-is-1.runpod.io --human-readable
   

5. Expected: 2025-10-28 10:49:07 17.3 MiB hyperopt_mamba2_demo

If GPU Utilization < 70%

1. Check actual batch size being used:

   grep "batch_size=" /workspace/logs/hyperopt_*.log
   

2. Verify --batch-size-max parameter in pod command:

   python3 -c "
   import requests, os
   from dotenv import load_dotenv
   load_dotenv('.env.runpod')
   api_key = os.getenv('RUNPOD_API_KEY')
   response = requests.get(
       'https://rest.runpod.io/v1/pods/qlql87w5avv1q1',
       headers={'Authorization': f'Bearer {api_key}'}
   )
   print(response.json()['dockerStartCmd'])
   "
   

3. Expected to see: --batch-size-max, 144

If Epoch Time > 13 min

1. Possible issue: Wrong GPU allocated (not RTX A4000)
2. Check GPU type:

   ssh root@qlql87w5avv1q1.ssh.runpod.io "nvidia-smi --query-gpu=name --format=csv,noheader"
   

3. If not RTX A4000 16GB, adjust --batch-size-max accordingly:
   • RTX A5000 24GB: --batch-size-max 216
   • Tesla V100 16GB: --batch-size-max 128
   • RTX 4090 24GB: --batch-size-max 216

Next Steps

1. Immediate (15-30 min):

   • Wait for pod to initialize (3-5 min)
   • SSH into pod and verify training started
   • Check first trial logs for feature normalization
   • Verify losses are < 1.0 (CRITICAL)
   • Update validation table with actual metrics

2. First Trial Complete (~10 hours):

   • Review trial 1 final metrics
   • Verify GPU utilization 85-92%
   • Confirm epoch time ~10 min (1.5× speedup)
   • Check if best params are reasonable

3. All Trials Complete (~5.3 hours × 30 = ~159 hours = 6.6 days):

   • Extract best hyperparameters
   • Compare best trial vs baseline
   • Update CLAUDE.md with new hyperparameters
   • Retrain production model with best params
   • Deploy to production

IMPORTANT: This is a multi-day experiment. Monitor periodically, don't need to watch continuously.

Cost Projection

Phase	Duration	Cost
Initialization	3-5 min	$0.02
First Trial (validation)	8-10 hours	$2.00-2.50
Remaining 29 Trials	~240 hours	$60.00
TOTAL	~10 days	$62.02-62.52

CRITICAL: This is MUCH more expensive than initially estimated ($1.33). The estimate was for a SINGLE trial (30 epochs), not 30 trials × 50 epochs each.

Recommendation:

1. Validate first trial succeeds (losses < 1.0)
2. Let 3-5 trials complete to verify convergence
3. If working well, let all 30 trials complete
4. Consider reducing trials to 10-15 if budget is tight (still get good hyperparameters)

Files Modified

1. /home/jgrusewski/Work/foxhunt/ml/src/hyperopt/adapters/mamba2.rs (line 475-505)

   • Added feature normalization
   • Added validation for zero variance features
   • Added logging for feature min/max/range

2. /home/jgrusewski/Work/foxhunt/target/release/examples/hyperopt_mamba2_demo (17.3 MiB)

   • Compiled with CUDA 12.9.1 + cuDNN 9
   • Stripped for size optimization
   • Uploaded to Runpod S3

3. /home/jgrusewski/Work/foxhunt/HYPEROPT_DEPLOYMENT_VALIDATION.md (this file)

   • Deployment validation report
   • Monitoring instructions
   • Success criteria

References

• Bug Analysis: HYPEROPT_LOSS_CALCULATION_BUG_ANALYSIS.md
• CLI Implementation: BATCH_SIZE_CLI_IMPLEMENTATION.md
• Deployment Script: scripts/runpod_deploy.py
• System Architecture: CLAUDE.md

---

Report Generated: 2025-10-28 10:55 UTC Next Update: After first trial validation (15-30 min)