foxhunt/AGENT_79_TFT_TUNING_QUICKSTART.md

Agent 79: TFT Optuna Tuning - Quick Start Guide

Status: ⚠️ READY AFTER FIX - TFT trainer needs 15-minute compilation fix Duration: 8-10 hours (30 trials, overnight run) Expected Best Hyperparameters: Learning rate 0.0003, Batch 64, Hidden 256, Heads 8

---

Current Blockers

Compilation Errors (3 errors)

File: /home/jgrusewski/Work/foxhunt/ml/src/trainers/tft.rs Lines: 709, 736, 756

Error:

error[E0599]: no method named `grad` found for reference `&Var`

Root Cause: Candle's Var type doesn't expose grad() method.

Fix (15 minutes):

// Comment out gradient monitoring methods (lines 709-760)
fn compute_gradient_norm(&self) -> f64 {
    0.0  // TODO: Implement when candle exposes grad()
}

fn clip_gradients(&mut self, _max_norm: f64) {
    // TODO: Implement when candle exposes grad()
    debug!("Gradient clipping not available in current candle version");
}

After fix, verify:

cargo build -p ml
cargo test -p ml test_tft_forward

---

Quick Commands

1. Fix TFT Compilation (15 min)

# Option 1: Manual fix (see error details above)
vi ml/src/trainers/tft.rs

# Option 2: Automated fix (if script exists)
./scripts/fix_tft_gradients.sh

# Verify compilation
cargo build -p ml

2. Update Tuning Config (5 min)

# Copy recommended config
cp TFT_TUNING_CONFIG_RECOMMENDED.yaml \
   services/ml_training_service/tuning_config.yaml

# Verify YAML syntax
python3 -c "import yaml; yaml.safe_load(open('services/ml_training_service/tuning_config.yaml'))"

3. Start Tuning Study (8-10 hours)

# Via TLI (recommended)
tli tune start \
  --model TFT \
  --trials 30 \
  --watch \
  --data-path test_data/real/databento/ml_training_small \
  --symbols 6E.FUT

# Direct gRPC (advanced)
grpcurl -d '{
  "model_type": "TFT",
  "n_trials": 30,
  "timeout_seconds": 36000,
  "hyperparameter_space": {...}
}' localhost:50054 ml_training.MLTrainingService/StartHyperparameterTuning

4. Monitor Progress

# Check status every 2 hours
tli tune status --job-id <uuid>

# Get current best
tli tune best --job-id <uuid>

# Cancel if needed
tli tune stop --job-id <uuid>

5. Cross-Validate (2-3 hours)

# Train with best hyperparameters on 6E.FUT
cargo run -p ml --example train_tft_dbn --release -- \
  --epochs 100 \
  --batch-size 64 \
  --learning-rate 0.0003 \
  --hidden-dim 256 \
  --num-attention-heads 8 \
  --lookback-window 60 \
  --forecast-horizon 10 \
  --data-path test_data/real/databento/ml_training_small \
  --output-dir ml/trained_models/production/tft_tuned

# Test on ES.FUT, NQ.FUT, ZN.FUT
./scripts/cross_validate_tft.sh ml/trained_models/production/tft_tuned/tft_final.safetensors

---

Expected Outcomes

Trial Duration Estimates

Configuration	Duration	Notes
30 epochs	8-10 min	Minimum for convergence
50 epochs	12-15 min	Recommended
100 epochs	18-22 min	Maximum (early stopping likely triggers)

With MedianPruner: 30-40% time savings on poor trials

Best Hyperparameters (Expected)

learning_rate: 0.0003        # Conservative for time-series
batch_size: 64               # Max for 4GB VRAM
hidden_dim: 256              # Max for 4GB VRAM
num_heads: 8                 # Standard attention heads
num_layers: 3                # Balance depth and memory
lookback_window: 60          # 1 hour context (1-min bars)
forecast_horizon: 10         # 10 minute predictions
dropout_rate: 0.15           # Mild regularization
early_stopping_patience: 20  # Conservative patience

Performance Targets

Metric	Target	Reasoning
Quantile Loss	0.15-0.25	Strong forecast accuracy
RMSE	0.008-0.012	1% prediction error
Sharpe Ratio	1.3-1.8	Excellent risk-adjusted returns
Trade Frequency	80-120	1.1-1.7% trade rate (7,223 bars)
Win Rate	55-60%	Consistent profitability
Max Drawdown	< 15%	Acceptable risk

---

Success Criteria

Minimum Viable (10% of trials)

• Quantile loss < 0.3
• Sharpe ratio > 1.0
• Validation loss within 20% of training loss
• Trade frequency 50-150 per 7,223 bars

Production-Ready (3% of trials)

• Quantile loss < 0.2
• Sharpe ratio > 1.5
• RMSE < 0.01
• Quantile coverage > 85%
• Cross-symbol Sharpe within 30%

Study Success

• ≥ 3 trials meet minimum viable
• ≥ 1 trial meets production-ready
• Best trial improves over baseline by 20%

---

Troubleshooting

GPU Memory Overflow

Symptom: CUDA OOM errors during training Fix:

# Reduce batch size
batch_size: 32  # Instead of 64

# Reduce hidden_dim
hidden_dim: 128  # Instead of 256

Poor Convergence

Symptom: All trials achieve loss > 0.5 Fix:

# Lower learning rate
learning_rate:
  low: 0.00005
  high: 0.0005

# Increase lookback window
lookback_window: [60, 90, 120]

Overfitting

Symptom: Training loss < 0.1, validation loss > 0.3 Fix:

# Increase dropout
dropout_rate:
  low: 0.1
  high: 0.4

# Reduce model capacity
hidden_dim: [64, 128]

Long Runtime

Symptom: 30 trials takes > 12 hours Fix:

# Reduce epochs
epochs:
  low: 20
  high: 50

# Increase pruning aggressiveness
median_pruner:
  n_warmup_steps: 20  # From 30

---

File Checklist

Created:

• ✅ AGENT_79_TFT_OPTUNA_TUNING_PLAN.md - Full analysis report (10,000 words)
• ✅ TFT_TUNING_CONFIG_RECOMMENDED.yaml - Recommended search space
• ✅ AGENT_79_TFT_TUNING_QUICKSTART.md - This quick start guide

Modified:

• ⏳ ml/src/trainers/tft.rs - Needs compilation fix (3 errors)
• ⏳ services/ml_training_service/tuning_config.yaml - Needs update (optional)

Required:

• ❌ No files from Agents 1-3 (unknown status)

---

Next Agent Tasks

Agent 80: Fix TFT Compilation (15 min)

1. Comment out compute_gradient_norm() (lines 709-716)
2. Comment out clip_gradients() (lines 736-760)
3. Remove gradient tracking from train_epoch() (lines 477-478, 502-510)
4. Verify: cargo build -p ml
5. Test: cargo test -p ml test_tft_forward

Agent 81: Run Tuning Study (8-10 hours)

1. Update tuning_config.yaml with recommended TFT config
2. Start tuning: tli tune start --model TFT --trials 30
3. Monitor every 2 hours: tli tune status
4. Document best hyperparameters
5. Run cross-validation on ES/NQ/ZN

---

References

• AGENT_79_TFT_OPTUNA_TUNING_PLAN.md - Full technical analysis
• AGENT_56_TFT_TRAINING_REPORT.md - Original TFT training failure
• AGENT_64_TFT_SHAPE_FIX.md - Broadcasting shape error fix
• AGENT_72_DBN_PARSER_FIX_REPORT.md - DBN data loading fix (7,223 bars)
• OPTUNA_TUNING_INTEGRATION_REPORT.md - Production Optuna infrastructure

---

Agent 79 Deliverables: ✅ COMPLETE

• ✅ Assessed TFT status (compilation errors identified)
• ✅ Configured 30-trial search space (memory-optimized)
• ✅ Documented execution plan (8-10 hour timeline)
• ✅ Created recommended tuning config (production-ready)

Next: Fix TFT compilation, then run 30-trial overnight study