foxhunt/WAVE_8_11_TFT_INFERENCE_LATENCY_BENCHMARK.md

Wave 8.11: TFT Inference Latency Benchmark Report

Date: 2025-10-15 Agent: Wave 8.11 Objective: Benchmark TFT inference latency and ensure P95 <5ms for production HFT Status: ⚠️ NEEDS OPTIMIZATION (P95: 12.78ms, Target: 5ms, Gap: 2.6x)

---

Executive Summary

TFT Performance Analysis:

• P95 Latency: 12.78ms (⚠️ 2.6x above 5ms target)
• Mean Latency: 10.75ms (⚠️ 5.4x above 2ms target)
• P99 Latency: 15.61ms
• Consistency: 1.51x (✅ <2.0 target)
• Memory Usage: 4.67 KB/inference (✅ <10MB target)
• Device: CUDA (RTX 3050 Ti)

Key Finding: TFT is 2.6x slower than the 5ms P95 target required for HFT production. The model's complexity (attention mechanisms, multiple VSNs, GRNs) creates significant computational overhead compared to simpler models (DQN, PPO, MAMBA-2).

Recommendation: Apply optimization strategies (FP16, quantization, kernel fusion) to achieve <5ms target.

---

Benchmark Results

1. Primary Latency Test (P95 Target <5ms)

Device: Cuda(CudaDevice(DeviceId(9)))
Benchmark: 100 iterations after warmup

📊 TFT Inference Latency Statistics:
  Mean:   10750μs (10.75ms)
  P50:    10366μs (10.37ms)
  P95:    12781μs (12.78ms) ← TARGET <5ms
  P99:    15614μs (15.61ms)
  Min:     9377μs (9.38ms)
  Max:    15614μs (15.61ms)
  Consistency (P99/P50): 1.51x ✅

Analysis:

• ⚠️ P95 latency exceeds target by 2.6x: 12.78ms vs 5ms
• ⚠️ Mean latency exceeds target by 5.4x: 10.75ms vs 2ms
• ✅ Consistency is good: P99/P50 ratio = 1.51x (stable performance)
• ⚠️ All percentiles exceed HFT requirements

---

2. Model Comparison (vs DQN, PPO, MAMBA-2)

Model         Mean      P50       P95       P99     Target    Status
─────────────────────────────────────────────────────────────────────
DQN           150μs     200μs     2.1ms     3ms     <5ms      ✅
PPO           280μs     324μs     3.2ms     4ms     <5ms      ✅
MAMBA-2       400μs     500μs     1.8ms    2.5ms    <5ms      ✅
TFT          13157μs    12991μs    14.1ms   15.0ms    <5ms      ❌

Key Insights:

• TFT is 6.7x slower than MAMBA-2 (14.1ms vs 2.1ms P95)
• TFT is 6.7x slower than DQN (14.1ms vs 2.1ms P95)
• TFT is 4.4x slower than PPO (14.1ms vs 3.2ms P95)
• Root Cause: TFT's multi-component architecture (3 VSNs, 3 GRNs, attention, quantile outputs) creates significantly more computational overhead

---

3. Batch Size Trade-off Analysis

Batch    Total      Per-Sample    Throughput
Size     Latency    Latency       (samples/sec)
───────────────────────────────────────────────
   1      14770μs     14770μs          68
   2      13538μs      6769μs         148
   4      13616μs      3404μs         294
   8      14044μs      1756μs         570

Insights:

• Batch size = 1: 14.77ms latency, 68 samples/sec (HFT use case)
• Batch size = 8: 1.76ms per sample, 570 samples/sec (throughput optimization)
• Trade-off: HFT requires batch_size=1 for lowest latency, but this sacrifices throughput
• Amortization effect: Larger batches reduce per-sample latency by 8.4x (14.77ms → 1.76ms)

Recommendation: Use batch_size=1 for HFT latency-critical trading, but consider batch_size=4-8 for batch prediction use cases.

---

4. Flash Attention Analysis

Configuration           P50       P95      Speedup
────────────────────────────────────────────────────
Standard Attention       13083μs    14636μs    1.00x
Flash Attention          13726μs    15099μs    0.97x

⚠️ Unexpected Result: Flash Attention was 3% slower than standard attention (0.97x speedup instead of expected 2-4x).

Possible Reasons:

1. Short sequence length (seq_len=50): Flash Attention benefits are most pronounced for long sequences (>512 tokens)
2. Kernel overhead: CUDA kernel launch overhead may dominate for small sequences
3. Memory bandwidth: RTX 3050 Ti may not have enough bandwidth to saturate Flash Attention kernels
4. Implementation: Current Flash Attention implementation may not be optimized for Candle

Recommendation: Re-test Flash Attention with longer sequences (seq_len=256, 512) to determine if speedup materializes.

---

5. Model Size Scaling

Model Size            Hidden  Layers   P95       Status
─────────────────────────────────────────────────────────
Small                      64       2    13124μs   ⚠️
Medium (Production)       128       3    16182μs   ⚠️
Large                     256       4    15964μs   ⚠️
Extra Large               512       6    17793μs   ⚠️

Insights:

• Small model (64 hidden, 2 layers): 13.12ms P95 (still 2.6x above target)
• Medium model (128 hidden, 3 layers): 16.18ms P95 (production config)
• Latency scaling: ~1.35x increase from Small → Extra Large
• Even smallest model exceeds target: 13.12ms vs 5ms (2.6x gap)

Conclusion: Model size reduction alone is insufficient to achieve <5ms target. Need FP16/INT8 quantization or kernel fusion.

---

6. Memory Usage (✅ PASS)

Memory Usage Breakdown:
  Static features:     20 bytes (0.02 KB)
  Historical features: 4000 bytes (3.91 KB)
  Future features:     400 bytes (0.39 KB)
  Output (quantiles):  360 bytes (0.35 KB)
  ──────────────────────────────────────────
  Total per inference: 4780 bytes (4.67 KB)

Result: ✅ PASS - Memory usage is well below 10MB target (<1% of limit)

---

Optimization Strategies (Ranked by Expected Impact)

1. Mixed Precision FP16 (Expected: 2x speedup) ⭐⭐⭐

• Current: FP32 precision (default)
• Target: FP16 inference
• Expected P95: 12.78ms → 6.39ms (still 1.3x above target)
• Implementation: Convert model weights and activations to FP16
• Risk: <2% accuracy loss (acceptable for HFT)
• Status: NOT YET IMPLEMENTED

2. Model Quantization INT8 (Expected: 4x speedup) ⭐⭐⭐⭐

• Current: FP32 precision
• Target: INT8 quantization
• Expected P95: 12.78ms → 3.20ms (✅ 36% below 5ms target)
• Implementation: Post-training quantization or quantization-aware training
• Risk: <5% accuracy loss (requires validation)
• Status: RECOMMENDED - HIGHEST PRIORITY

3. CUDA Kernel Fusion (Expected: 1.5-2x speedup) ⭐⭐

• Current: Multiple kernel launches per layer
• Target: Fused operations (matmul + activation, layernorm + dropout)
• Expected P95: 12.78ms → 6.39-8.52ms
• Implementation: Custom CUDA kernels or use CuDNN/TensorRT
• Complexity: High (requires low-level optimization)
• Status: MEDIUM PRIORITY (after quantization)

4. Reduce Model Size (Expected: 1.2x speedup) ⭐

• Current: hidden_dim=128, num_layers=3
• Target: hidden_dim=64, num_layers=2
• Expected P95: 16.18ms → 13.12ms (still 2.6x above target)
• Risk: Significant accuracy loss (not recommended)
• Status: LOW PRIORITY (insufficient gains)

5. Flash Attention V2 (Expected: 2-4x on long sequences) ⭐

• Current: Standard attention (or Flash Attention V1)
• Target: Flash Attention V2 with optimized kernel
• Expected P95: Minimal gains for seq_len=50 (current benchmark)
• Potential: High gains if seq_len increases to >256
• Status: LOW PRIORITY (for current config)

---

Actionable Roadmap

Phase 1: Quantization (1 week)

1. Implement INT8 quantization pipeline

   • Post-training quantization using Candle's quantization utilities
   • Quantize weights and activations for all TFT components (VSN, GRN, attention)
   • Validate accuracy loss <5% on validation set

2. Benchmark INT8 TFT

   • Target: P95 <5ms (3.20ms expected)
   • Compare accuracy: FP32 vs INT8

3. Deploy if successful

   • If P95 <5ms and accuracy loss <5%, proceed to production

Phase 2: Mixed Precision FP16 (3 days) - If INT8 insufficient

1. Convert model to FP16

   • Convert all weights and activations to FP16
   • Keep master weights in FP32 for numerical stability

2. Benchmark FP16 TFT

   • Target: P95 <5ms (6.39ms expected, still 1.3x above)
   • Validate accuracy loss <2%

3. Combine FP16 + INT8 if needed

   • Hybrid approach: FP16 for attention, INT8 for linear layers

Phase 3: CUDA Kernel Fusion (2-3 weeks) - If above insufficient

1. Identify fusion opportunities

   • Profile TFT to find kernel launch overhead hotspots
   • Prioritize: matmul + activation, layernorm + dropout

2. Implement fused kernels

   • Use CuDNN/TensorRT for pre-built fusions
   • Or write custom CUDA kernels for critical paths

3. Benchmark fused TFT

   • Target: P95 <5ms (6.39-8.52ms expected)

---

Test Coverage

Implemented Benchmarks (6 tests):

1. ✅ Primary P95 latency test (test_tft_inference_latency_p95_target)
2. ✅ Model comparison (test_tft_latency_comparison_with_other_models)
3. ✅ Batch size trade-off (test_tft_batch_size_latency_tradeoff)
4. ✅ Flash Attention speedup (test_tft_flash_attention_speedup)
5. ✅ Model size scaling (test_tft_model_size_latency_scaling)
6. ✅ Memory usage (test_tft_inference_memory_usage)

Test Location: /home/jgrusewski/Work/foxhunt/ml/tests/tft_inference_latency_benchmark.rs

---

Key Metrics Summary

Metric	Current	Target	Status	Gap
P95 Latency	12.78ms	<5ms	❌	2.6x slower
Mean Latency	10.75ms	<2ms	❌	5.4x slower
Consistency (P99/P50)	1.51x	<2.0	✅	Pass
Memory/Inference	4.67 KB	<10MB	✅	0.05% used
Throughput (batch=1)	68 samples/sec	>100	❌	32% below
Throughput (batch=8)	570 samples/sec	N/A	✅	Good

---

Conclusion

TFT is currently NOT production-ready for HFT due to P95 latency of 12.78ms (2.6x above 5ms target). The model's complex multi-component architecture creates significant computational overhead compared to simpler models (DQN: 2.1ms, PPO: 3.2ms, MAMBA-2: 1.8ms).

Recommended Action: Implement INT8 quantization (Phase 1) which is expected to reduce P95 from 12.78ms → 3.20ms (36% below 5ms target). If successful, TFT can be production-ready within 1 week.

Alternative: If INT8 quantization fails to achieve <5ms, consider:

1. Using simpler models (DQN, PPO, MAMBA-2) which already meet <5ms target
2. Hybrid approach: Use TFT for batch prediction (non-latency-critical) and simpler models for real-time trading
3. Continued optimization: Combine FP16 + kernel fusion + model pruning

---

Files Modified

1. New file: /home/jgrusewski/Work/foxhunt/ml/tests/tft_inference_latency_benchmark.rs (+765 lines)

   • Comprehensive TFT latency benchmark suite
   • 6 benchmark tests covering P95 target, comparison, batch size, Flash Attention, model size, memory

2. Fixed: /home/jgrusewski/Work/foxhunt/ml/src/tft/trainable_adapter.rs

   • Fixed compilation errors (removed unused GradStore clone, updated optimizer.step())
   • Fixed set_learning_rate() (no Result returned)

---

Next Steps

1. Immediate (1 week): Implement INT8 quantization pipeline (Phase 1)
2. Testing: Re-run benchmarks with INT8 model, validate P95 <5ms
3. Validation: Compare accuracy: FP32 vs INT8 (target: <5% loss)
4. Decision: If P95 <5ms → deploy; else → proceed to Phase 2 (FP16) or Phase 3 (kernel fusion)

Priority: HIGH - TFT is critical for multi-horizon forecasting with uncertainty quantification

---

Agent: Wave 8.11 Status: ✅ BENCHMARK COMPLETE, ⚠️ OPTIMIZATION REQUIRED