foxhunt/MAMBA2_WEIGHT_DECAY_FIX_VALIDATION.md

MAMBA-2 Weight Decay Fix - Validation Monitoring

Date: 2025-10-27 Pod ID: 202o2kkocnu5wz GPU: RTX 4090 (24GB VRAM) Datacenter: EUR-IS-1 Cost: $0.59/hr Training Duration: ~93 minutes (1.86 min/epoch × 50 epochs) Total Cost: ~$0.91

---

Fix Applied

Bug: Weight decay configured (1e-4) but NEVER applied in Adam optimizer Location: ml/src/mamba/mod.rs:1979-1990 Root Cause: Adam optimizer used raw gradients without weight decay L2 penalty Impact: SSM matrices (~43k parameters) trained without regularization → severe overfitting

Fix (lines 1979-1998):

// P0-CRITICAL FIX (Agent 280): Apply weight decay before Adam momentum update
let effective_grad = if self.config.weight_decay > 0.0 {
    let wd_term = (var.as_tensor() * self.config.weight_decay)?;
    (grad + wd_term)?
} else {
    grad.clone()
};

// Adam update equations (use effective_grad with weight decay)
let m_new = ((&m * beta1)? + (&effective_grad * (1.0 - beta1))?)?;
let v_new = ((&v * beta2)? + (effective_grad.sqr()? * (1.0 - beta2))?)?;

---

Training Configuration

/runpod-volume/binaries/train_mamba2_parquet \
  --parquet-file /runpod-volume/test_data/ES_FUT_180d.parquet \
  --epochs 50 \
  --batch-size 512 \
  --learning-rate 0.00005 \
  --use-gpu

Dataset: ES_FUT_180d.parquet (21,600 bars, 80/20 split) Optimizer: Adam (beta1=0.9, beta2=0.999, weight_decay=1e-4) LR Schedule: Cosine annealing with warmup Binary: 20,738,736 bytes (uploaded Oct 27 13:21:39)

---

Expected Results

BEFORE FIX (Broken - Weight Decay NOT Applied)

E0:  train=--, val=27.6M (BEST - initialization) ✅
E5:  train=19.4M, val=29.8M (+8.0% overfitting)
E10: train=18.9M, val=31.5M (+14.1% overfitting)
E15: train=14.8M, val=32.1M (+16.3% overfitting) 🔴
E15: train dropped 17% in ONE epoch (17.8M → 14.8M)

Overfitting Ratio: 2.17x (CRITICAL)

Problem: E0 initialization BETTER than ANY trained epoch

AFTER FIX (Expected - Weight Decay Applied)

E0:  train=--, val=27.6M (initialization)
E5:  train=22.0M, val=25.5M (-7.6% improvement) ✅
E10: train=19.5M, val=23.8M (-13.8% improvement) ✅
E15: train=18.2M, val=23.5M (-14.9% improvement) ✅ BEST
E20: train=17.8M, val=23.6M (slight overfit, early stopping)

Overfitting Ratio: 1.3x (HEALTHY)

Key Differences:

• ✅ Best val_loss at E10-E15 (not E0)
• ✅ 50-70% reduction in overfitting (32.1M → 23.5M, -27%)
• ✅ Training converges to optimal point
• ✅ Weight decay prevents parameter explosion

---

Monitoring Checkpoints

1. Pod Initialization (0-3 minutes)

Status: 🟡 IN PROGRESS (waiting for pod to initialize)

Expected:

• ✅ Pod created: 202o2kkocnu5wz
• ✅ Docker image loaded: jgrusewski/foxhunt:latest
• ✅ Network volume mounted: /runpod-volume/
• ⏳ CUDA device detected: RTX 4090
• ⏳ Binary executable permission set
• ⏳ Training process started

SSH Command:

ssh root@202o2kkocnu5wz.ssh.runpod.io

Verification Commands:

# Check GPU
nvidia-smi

# Check binary
ls -lh /runpod-volume/binaries/train_mamba2_parquet

# Check training logs
tail -f /workspace/training.log

# Check process
ps aux | grep train_mamba2

2. Training Start (3-8 minutes)

Status: ⏳ PENDING

Expected E0-E5 Losses:

E0: train ≈ 85M, val ≈ 82M (random initialization)
E1: train ≈ 78M, val ≈ 75M
E2: train ≈ 72M, val ≈ 70M
E3: train ≈ 68M, val ≈ 66M
E4: train ≈ 64M, val ≈ 62M
E5: train ≈ 61M, val ≈ 59M

Validation Criteria:

• ✅ Training loss decreases smoothly
• ✅ Validation loss tracks training loss
• ✅ No NaN/Inf values
• ✅ GPU memory stable (~164MB)

3. E10-E15 (20-30 minutes) CRITICAL VALIDATION WINDOW

Status: ⏳ PENDING

PRIMARY OBJECTIVE: Verify overfitting is eliminated

Expected Behavior:

E10: val_loss ≈ 23-26M (smooth decline from E0's 27.6M) ✅
E11: val_loss ≈ 22-25M (smooth decline, NO spike) ✅
E12: val_loss ≈ 22-24M
E13: val_loss ≈ 21-24M
E14: val_loss ≈ 21-23M
E15: val_loss ≈ 20-23M (BETTER than broken 32.1M) ✅

SUCCESS CRITERIA:

• ✅ E15 val_loss < 26M (vs broken 32.1M, -19% minimum improvement)
• ✅ Best val_loss at E10-E20 (NOT at E0)
• ✅ Overfitting ratio < 1.5x (vs broken 2.17x)

Red Flags (if seen, IMMEDIATE INVESTIGATION):

• ❌ E15 val_loss > 30M → Weight decay fix NOT working
• ❌ E0 still best val_loss → Model still overfitting
• ❌ NaN/Inf at any epoch → Numerical instability

4. E30 (55 minutes)

Status: ⏳ PENDING

Expected:

• ✅ Warmup phase ends (LR reaches 5e-5)
• ✅ Training continues smoothly
• ✅ Validation loss ≈ 20-22M

5. E50 (93 minutes)

Status: ⏳ PENDING

Expected:

• ✅ Training completes successfully
• ✅ Final validation loss ≈ 18-21M (10-15% improvement from E0)
• ✅ Model checkpoints saved to /runpod-volume/models/
• ✅ Pod auto-terminates (entrypoint-self-terminate.sh)

---

Success Metrics

PRIMARY (Weight Decay Fix Validation)

• ✅ Best val_loss at E10-E20 (NOT E0)
• ✅ E15 val_loss < 26M (vs broken 32.1M, -19% minimum)
• ✅ Overfitting ratio < 1.5x (vs broken 2.17x)

SECONDARY (Model Convergence)

• ✅ Training loss decreases smoothly
• ✅ Validation loss decreases (not increases)
• ✅ No NaN/Inf values
• ✅ Final val_loss ≈ 18-21M (10-15% improvement from E0)

TERTIARY (Training Stability)

• ✅ No crashes/OOM errors
• ✅ GPU memory stable (<500MB)
• ✅ Checkpoints saved successfully

---

Validation Timeline

00:00 - Pod deployed
00:03 - SSH into pod, verify training started
00:08 - Check E0-E5 logs, verify smooth decline
00:20 - CRITICAL: Monitor E10 logs
00:22 - CRITICAL: Monitor E11 logs (no spike expected)
00:28 - CRITICAL: Monitor E15 logs (must be < 26M)
00:55 - Check E30 logs (warmup complete)
01:33 - Training completes, verify final results
01:35 - Download logs and checkpoints
01:40 - Update CLAUDE.md with results

---

Data Collection

Logs to Save

1. Full training logs: /workspace/training.log → save locally
2. E10-E15 excerpt: Extract and save to final report
3. GPU metrics: nvidia-smi snapshots at E0, E10, E15, E30, E50
4. Checkpoints: Download E10, E15, E50 from /runpod-volume/models/

Metrics to Extract

• E0-E50 train/val losses (CSV format)
• E10-E15 validation loss deltas (%)
• Overfitting ratio at E15: train_loss / val_loss
• Final improvement: (val_E0 - val_E50) / val_E0 * 100

---

Failure Scenarios & Actions

Scenario 1: E15 val_loss > 30M (Weight decay NOT working)

Cause: Fix not applied correctly or binary mismatch

Action:

1. Verify binary timestamp: ls -lh /runpod-volume/binaries/train_mamba2_parquet
2. Check binary SHA256 vs local
3. Review weight decay code in ml/src/mamba/mod.rs:1979-1998
4. Re-upload fixed binary and restart training

Scenario 2: E0 still best val_loss (Model still overfitting)

Cause: Weight decay too weak or other overfitting source

Action:

1. Extract weight decay value from logs
2. Verify weight decay = 1e-4 in training config
3. Consider increasing weight decay to 1e-3
4. Check if dropout/other regularization needed

Scenario 3: NaN/Inf values appear

Cause: Numerical instability from weight decay fix

Action:

1. Check gradient norms (should be clipped to 1.0)
2. Verify Adam epsilon value (1e-8)
3. Check if weight decay term causes explosion
4. Consider gradient scaling or mixed precision

Scenario 4: E15 val_loss 26-30M (Partial improvement)

Cause: Weight decay working but not optimal

Action:

1. ACCEPT RESULT (partial improvement is success)
2. Document 10-20% improvement vs broken version
3. Consider tuning weight decay for future runs
4. Proceed to production with current fix

---

Next Steps After Validation

If E15 val_loss < 26M (SUCCESS ✅)

1. Update CLAUDE.md: Mark MAMBA-2 as "✅ Weight Decay Fixed"
2. Create Final Report: MAMBA2_WEIGHT_DECAY_FIX_FINAL_REPORT.md
3. Commit Changes: Git commit with weight decay fix
4. Proceed to Production: All models certified, ready for deployment

If E15 val_loss 26-30M (PARTIAL SUCCESS ⚠️)

1. Document Results: Partial improvement achieved
2. Tune Weight Decay: Test 1e-3, 5e-4 values
3. Defer Production: Optimize before deployment
4. Continue Investigation: Other regularization techniques

If E15 val_loss > 30M (FAILURE ❌)

1. Binary Verification: Confirm correct binary deployed
2. Code Review: Re-verify weight decay implementation
3. Emergency Debug Session: Deep dive investigation
4. Block Production: Do not proceed until fixed

---

Status

Current Phase: 🟡 Pod Initialization (0-3 minutes) Next Action: SSH into pod, verify training started Critical Window: E10-E15 (20-30 minutes from now)

---

Report End