foxhunt/LIQUID_NN_TUNING_GUIDE.md

Liquid Neural Network Hyperparameter Tuning Guide

Mission: Optimize Liquid NN for ultra-low latency HFT with ODE integration analysis

Expected Duration: 4-6 hours (30 trials)

Status: Ready to Execute

---

Executive Summary

This guide documents the hyperparameter tuning process for Foxhunt's new Liquid Neural Network (LNN) model - a continuous-time neural ODE architecture designed for ultra-low latency inference (<100μs) and market regime adaptation.

Key Innovations

1. Continuous-Time Dynamics: ODE-based state evolution (vs discrete LSTM/DQN)
2. Adaptive Time Constants: Market volatility-aware behavior
3. Sparse Connectivity: Pruned connections for faster inference
4. Multiple ODE Solvers: Euler (fast), RK4 (accurate), Adaptive (regime-aware)

---

Quick Start

Prerequisites

# 1. Ensure ML Training Service is running
cargo run -p ml_training_service --release &

# 2. Verify GPU availability (optional but recommended)
nvidia-smi

# 3. Check data files
ls -lh test_data/*.dbn.zst

Execute Tuning

# Run 30-trial hyperparameter optimization
./run_liquid_nn_tuning.sh

# Monitor progress (tuning continues in background)
tail -f ml/trained_models/tuning/liquid_nn/tuning_execution.log

---

Configuration Details

Search Space (tuning_config.yaml)

Core Architecture

• Learning Rate: [0.0001, 0.001, 0.01] (3 choices)
• Batch Size: [32, 64, 128] (3 choices)
• Hidden Dim: [64, 128, 256] (3 choices)
• Num Layers: [1, 2, 3] (3 choices)

ODE Integration (Critical for Continuous-Time)

• ODE Steps: [3, 5, 10, 20] (4 choices)
  • 3 steps: Fast but less accurate
  • 5 steps: Balanced (recommended baseline)
  • 10 steps: Higher accuracy, moderate cost
  • 20 steps: Maximum accuracy, slowest
• Solver Type: [Euler, RK4, Adaptive] (3 choices)
  • Euler: 1st order, fastest, ~O(dt) error
  • RK4: 4th order, slower, ~O(dt^4) error
  • Adaptive: Switches based on market regime
• Default dt: [0.001 - 0.1] (logarithmic scale)

Network Structure

• Sparsity Level: [0.5, 0.7, 0.9] (3 choices)
  • 0.5: 50% connections pruned
  • 0.7: 70% connections pruned (recommended)
  • 0.9: 90% connections pruned (fastest)
• Network Type: [LTC, CfC, Mixed] (3 choices)
  • LTC: Liquid Time-constant neurons
  • CfC: Closed-form Continuous-time neurons
  • Mixed: Combination of LTC and CfC layers

Time Constants (τ)

• Base τ: [0.01, 0.1, 1.0] (3 choices)
• τ Min: [0.001 - 0.05] (log scale)
• τ Max: [0.5 - 5.0] (log scale)
• Adaptive τ: [true, false] (volatility-aware adaptation)

Activation & Regularization

• Cell Activation: [Tanh, Sigmoid, ReLU]
• Output Activation: [Linear, Tanh, Sigmoid]
• Dropout Rate: [0.0 - 0.3]
• L2 Regularization: [0.00001 - 0.001] (log scale)

Adaptive Features

• Market Regime Adaptation: [true, false]
  • Adjusts time step (dt) based on volatility
• Early Stopping Patience: [5, 10, 15, 20]

Combined Objective Function

objective = sharpe_ratio - 0.1 * log(inference_ms)

Rationale:

• Sharpe Ratio: Primary metric (risk-adjusted returns)
• Inference Time Penalty: Ensures HFT latency requirements
• Weight 0.1: Balanced tradeoff (logarithmic penalty)

Example Calculation:

• Sharpe Ratio = 1.5
• Inference Time = 0.08 ms (80 μs)
• Objective = 1.5 - 0.1 × log(0.08) = 1.5 - 0.1 × (-2.53) = 1.5 + 0.253 = 1.753

Target:

• Sharpe Ratio: >1.5
• Inference Time: <100 μs (0.1 ms)
• Combined Score: >1.4

---

Execution Workflow

Phase 1: Pre-flight Checks (2 minutes)

✓ Check tuning_config.yaml exists
✓ Verify LIQUID configuration
✓ Detect GPU (RTX 3050 Ti CUDA)
✓ Confirm ML Training Service running
✓ Validate data files (6E.FUT, ZN.FUT, ES.FUT, NQ.FUT)

Phase 2: Tuning Execution (4-6 hours)

Trial   1/30: [Sampling hyperparameters...]
  • Learning Rate: 0.001
  • Batch Size: 64
  • Hidden Dim: 128
  • ODE Steps: 5
  • Solver Type: Euler
  • Sparsity Level: 0.7
  • Network Type: LTC
  [Training... 50 epochs]
  Result: Sharpe=1.42, Inference=95μs, Combined=1.45

Trial   2/30: [TPE sampling based on Trial 1...]
  [...]

[Progress bar: ============================> 50% (15/30 trials)]
ETA: 2:15:00 remaining

Phase 3: Result Analysis (10 minutes)

Best Trial: 23/30
Best Hyperparameters:
  • Learning Rate: 0.001
  • Batch Size: 64
  • Hidden Dim: 256
  • ODE Steps: 10
  • Solver Type: RK4
  • Sparsity Level: 0.7
  • Network Type: CfC
  • Time Constant τ: 0.1
  • Adaptive τ: true

Performance:
  • Sharpe Ratio: 1.58
  • Inference Time: 87 μs
  • Combined Score: 1.61

Improvement over baseline: +12.4%

---

Analysis Framework

1. ODE Integration Analysis

Objective: Find optimal balance between accuracy and speed

Metrics to Compare

• Euler Solver: Fastest, suitable for low-volatility regimes
• RK4 Solver: Most accurate, higher computational cost
• Adaptive Solver: Best of both worlds

Expected Results Table

Solver	ODE Steps	Sharpe Ratio	Inference (μs)	Combined Score
Euler	3	1.35	45	1.45
Euler	5	1.42	65	1.47
RK4	5	1.48	110	1.45
RK4	10	1.55	180	1.42
Adaptive	5	1.52	85	1.57

Key Insight: Adaptive solver likely provides best combined score by dynamically switching based on market conditions.

2. Sparsity Level Analysis

Objective: Determine optimal connection pruning for inference speed

Tradeoff Analysis

Sparsity	Parameters	Memory (KB)	Inference (μs)	Sharpe Ratio	Combined Score
0.5	65,536	256	120	1.58	1.55
0.7	32,768	128	85	1.52	1.57
0.9	8,192	32	55	1.38	1.47

Key Insight: 0.7 sparsity level likely optimal - balances accuracy (1.52 Sharpe) with speed (85 μs).

3. Continuous-Time Advantage Analysis

Objective: Validate superiority over discrete-time models (LSTM, DQN)

Model Comparison

Model	Architecture	Inference (μs)	Sharpe Ratio	Training Time	Adaptability
LSTM	Discrete recurrent	500-1000	1.35	Fast	Low
DQN	Discrete RL	100-200	1.42	Moderate	Medium
Liquid NN	Continuous ODE	<100	>1.5	Slow	High

Key Advantages:

1. Speed: Sparse connectivity → 2-10x faster than LSTM
2. Accuracy: ODE integration → smooth continuous dynamics
3. Adaptability: Time constant modulation → regime-aware behavior

4. Market Regime Adaptation

Time Step Adjustment Based on Volatility:

• Normal: dt = 0.01 (base time constant)
• Sideways: dt = 0.005 (slower adaptation)
• Trending: dt = 0.02 (faster adaptation)
• Bull/Bear: dt = 0.0025 (high-frequency updates)
• Crisis: dt = 0.00125 (ultra-fast response)

Expected Impact: 5-10% improvement in Sharpe ratio during volatile periods.

---

Output Files

Generated Artifacts

ml/trained_models/tuning/liquid_nn/
├── tuning_execution.log              # Full execution log
├── job_id.txt                         # Tuning job UUID
├── best_hyperparameters.txt          # Best trial results
├── LIQUID_NN_TUNING_SUMMARY.md       # Comprehensive report
├── checkpoints/                       # Model checkpoints per trial
│   ├── trial_01_epoch_50.safetensors
│   ├── trial_02_epoch_50.safetensors
│   └── ...
├── plots/                             # Visualization plots
│   ├── ode_integration_comparison.png
│   ├── sparsity_performance.png
│   ├── inference_time_distribution.png
│   └── sharpe_ratio_progression.png
└── analysis/                          # Detailed analyses
    ├── ode_integration_analysis.md
    ├── sparsity_analysis.md
    └── continuous_time_advantage.md

Summary Report Structure

# Liquid NN Tuning Summary

## Best Hyperparameters
[Full configuration]

## ODE Integration Analysis
- Solver comparison (Euler vs RK4 vs Adaptive)
- Accuracy vs speed tradeoff
- Optimal ODE steps

## Sparsity Analysis
- Connection pruning impact
- Inference speed vs accuracy
- Memory footprint

## Continuous-Time Advantage
- Comparison with LSTM/DQN
- Regime adaptation analysis
- HFT latency validation

## Production Recommendations
[Deployment configuration]

---

Troubleshooting

Common Issues

1. GPU Memory Error

Error: CUDA out of memory

Solution: Reduce batch size in tuning_config.yaml or use CPU mode.

2. gRPC Connection Failed

Error: Failed to connect to localhost:50054

Solution: Start ML Training Service:

cargo run -p ml_training_service --release &

3. Missing Data Files

Warning: Missing data file: ES.FUT

Solution: Download DBN files or use available symbols only.

4. Tuning Timeout

Error: Trial timeout after 60 minutes

Solution: Increase max_epochs_per_trial or reduce epochs in tuning_config.yaml.

---

Next Steps

After Tuning Completes

1. Review Best Hyperparameters

   cat ml/trained_models/tuning/liquid_nn/best_hyperparameters.txt
   

2. Analyze ODE Integration

   cat ml/trained_models/tuning/liquid_nn/analysis/ode_integration_analysis.md
   

3. Validate Inference Speed

   cargo run -p ml --example benchmark_liquid_nn --release
   

4. Production Training

   # Use best hyperparameters for full 100-epoch training
   cargo run -p ml_training_service --example train_liquid_nn_production
   

5. Backtesting

   # Test with best model on historical data
   cargo run -p backtesting_service --example test_liquid_nn_strategy
   

---

Technical Deep Dive

ODE Integration Mathematics

Euler Method (1st Order)

x(t + dt) = x(t) + dt × f(x(t), t)

• Error: O(dt)
• Operations: 1 function evaluation per step
• Use Case: Low volatility, speed-critical

RK4 Method (4th Order)

k1 = f(x(t), t)
k2 = f(x(t) + dt/2 × k1, t + dt/2)
k3 = f(x(t) + dt/2 × k2, t + dt/2)
k4 = f(x(t) + dt × k3, t + dt)
x(t + dt) = x(t) + dt/6 × (k1 + 2k2 + 2k3 + k4)

• Error: O(dt^4)
• Operations: 4 function evaluations per step
• Use Case: High accuracy required

Adaptive Method

if market_regime in [Bull, Bear, Crisis]:
    solver = RK4  # High accuracy for volatile markets
else:
    solver = Euler  # Fast for normal conditions

Sparsity Implementation

Connection Pruning Strategy:

// Prune weights below threshold
let threshold = percentile(weights, sparsity_level);
for weight in weights {
    if abs(weight) < threshold {
        weight = 0.0;
    }
}

// Inference with sparse matrix multiplication
output = sparse_matmul(sparse_weights, input);

Memory Benefits:

• 0.5 sparsity: 50% memory reduction
• 0.7 sparsity: 70% memory reduction
• 0.9 sparsity: 90% memory reduction

Inference Benefits:

• Fewer multiplications → faster compute
• Better cache locality → reduced memory bandwidth
• Suitable for fixed-point arithmetic

---

Performance Expectations

Baseline Targets

Metric	Target	Rationale
Sharpe Ratio	>1.5	Risk-adjusted returns superiority
Inference Time	<100 μs	HFT latency requirement
Combined Score	>1.4	sharpe - 0.1×log(inference_ms)
Training Time/Trial	8-12 min	50 epochs × 10-15 sec/epoch
Total Tuning Duration	4-6 hours	30 trials × 8-12 min/trial
GPU Memory Usage	<3.5 GB	RTX 3050 Ti 4GB constraint

Success Criteria

✅ Minimum Acceptable:

• Sharpe Ratio: 1.3
• Inference Time: 150 μs
• Combined Score: 1.2

✅ Target Performance:

• Sharpe Ratio: 1.5
• Inference Time: 100 μs
• Combined Score: 1.4

✅ Stretch Goal:

• Sharpe Ratio: 1.8
• Inference Time: 80 μs
• Combined Score: 1.75

---

References

Papers & Research

1. Liquid Time-constant Networks (Hasani et al., 2021)

   • https://arxiv.org/abs/2006.04439
   • Introduces LTC neurons with adaptive time constants

2. Neural Ordinary Differential Equations (Chen et al., 2018)

   • https://arxiv.org/abs/1806.07366
   • Foundational work on continuous-time neural networks

3. Closed-form Continuous-time Neural Networks (Hasani et al., 2022)

   • https://www.nature.com/articles/s42256-022-00556-7
   • CfC neurons with closed-form solutions

Foxhunt Documentation

• CLAUDE.md: System architecture and ML infrastructure
• ML_TRAINING_ROADMAP.md: 4-6 week training plan
• GPU_TRAINING_BENCHMARK.md: RTX 3050 Ti performance benchmarks
• TESTING_PLAN.md: ML model validation strategy

---

Conclusion

This tuning campaign represents a significant milestone in Foxhunt's ML evolution:

1. Innovation: First continuous-time neural ODE model in HFT system
2. Performance: Ultra-low latency (<100μs) with high accuracy (Sharpe >1.5)
3. Adaptability: Market regime-aware behavior via time constant modulation
4. Efficiency: Sparse connectivity for faster inference and lower memory

Expected Outcome: Production-ready Liquid NN hyperparameters optimized for ODE integration, sparsity, and inference speed - ready for 100-epoch production training and live trading deployment.

---

Status: Ready to Execute Command: ./run_liquid_nn_tuning.sh Duration: 4-6 hours Next Milestone: Production training with optimized hyperparameters