foxhunt/LIQUID_NN_TUNING_GUIDE.md

# Liquid Neural Network Hyperparameter Tuning Guide

**Mission**: Optimize Liquid NN for ultra-low latency HFT with ODE integration analysis

**Expected Duration**: 4-6 hours (30 trials)

**Status**: Ready to Execute

---

## Executive Summary

This guide documents the hyperparameter tuning process for Foxhunt's new Liquid Neural Network (LNN) model - a continuous-time neural ODE architecture designed for ultra-low latency inference (<100μs) and market regime adaptation.

### Key Innovations

1. **Continuous-Time Dynamics**: ODE-based state evolution (vs discrete LSTM/DQN)
2. **Adaptive Time Constants**: Market volatility-aware behavior
3. **Sparse Connectivity**: Pruned connections for faster inference
4. **Multiple ODE Solvers**: Euler (fast), RK4 (accurate), Adaptive (regime-aware)

---

## Quick Start

### Prerequisites

```bash
# 1. Ensure ML Training Service is running
cargo run -p ml_training_service --release &

# 2. Verify GPU availability (optional but recommended)
nvidia-smi

# 3. Check data files
ls -lh test_data/*.dbn.zst
```

### Execute Tuning

```bash
# Run 30-trial hyperparameter optimization
./run_liquid_nn_tuning.sh

# Monitor progress (tuning continues in background)
tail -f ml/trained_models/tuning/liquid_nn/tuning_execution.log
```

---

## Configuration Details

### Search Space (tuning_config.yaml)

#### Core Architecture
- **Learning Rate**: [0.0001, 0.001, 0.01] (3 choices)
- **Batch Size**: [32, 64, 128] (3 choices)
- **Hidden Dim**: [64, 128, 256] (3 choices)
- **Num Layers**: [1, 2, 3] (3 choices)

#### ODE Integration (Critical for Continuous-Time)
- **ODE Steps**: [3, 5, 10, 20] (4 choices)
  - 3 steps: Fast but less accurate
  - 5 steps: Balanced (recommended baseline)
  - 10 steps: Higher accuracy, moderate cost
  - 20 steps: Maximum accuracy, slowest
- **Solver Type**: [Euler, RK4, Adaptive] (3 choices)
  - **Euler**: 1st order, fastest, ~O(dt) error
  - **RK4**: 4th order, slower, ~O(dt^4) error
  - **Adaptive**: Switches based on market regime
- **Default dt**: [0.001 - 0.1] (logarithmic scale)

#### Network Structure
- **Sparsity Level**: [0.5, 0.7, 0.9] (3 choices)
  - 0.5: 50% connections pruned
  - 0.7: 70% connections pruned (recommended)
  - 0.9: 90% connections pruned (fastest)
- **Network Type**: [LTC, CfC, Mixed] (3 choices)
  - **LTC**: Liquid Time-constant neurons
  - **CfC**: Closed-form Continuous-time neurons
  - **Mixed**: Combination of LTC and CfC layers

#### Time Constants (τ)
- **Base τ**: [0.01, 0.1, 1.0] (3 choices)
- **τ Min**: [0.001 - 0.05] (log scale)
- **τ Max**: [0.5 - 5.0] (log scale)
- **Adaptive τ**: [true, false] (volatility-aware adaptation)

#### Activation & Regularization
- **Cell Activation**: [Tanh, Sigmoid, ReLU]
- **Output Activation**: [Linear, Tanh, Sigmoid]
- **Dropout Rate**: [0.0 - 0.3]
- **L2 Regularization**: [0.00001 - 0.001] (log scale)

#### Adaptive Features
- **Market Regime Adaptation**: [true, false]
  - Adjusts time step (dt) based on volatility
- **Early Stopping Patience**: [5, 10, 15, 20]

### Combined Objective Function

```python
objective = sharpe_ratio - 0.1 * log(inference_ms)
```

**Rationale**:
- **Sharpe Ratio**: Primary metric (risk-adjusted returns)
- **Inference Time Penalty**: Ensures HFT latency requirements
- **Weight 0.1**: Balanced tradeoff (logarithmic penalty)

**Example Calculation**:
- Sharpe Ratio = 1.5
- Inference Time = 0.08 ms (80 μs)
- Objective = 1.5 - 0.1 × log(0.08) = 1.5 - 0.1 × (-2.53) = 1.5 + 0.253 = **1.753**

**Target**:
- Sharpe Ratio: >1.5
- Inference Time: <100 μs (0.1 ms)
- Combined Score: >1.4

---

## Execution Workflow

### Phase 1: Pre-flight Checks (2 minutes)

```
✓ Check tuning_config.yaml exists
✓ Verify LIQUID configuration
✓ Detect GPU (RTX 3050 Ti CUDA)
✓ Confirm ML Training Service running
✓ Validate data files (6E.FUT, ZN.FUT, ES.FUT, NQ.FUT)
```

### Phase 2: Tuning Execution (4-6 hours)

```
Trial   1/30: [Sampling hyperparameters...]
  • Learning Rate: 0.001
  • Batch Size: 64
  • Hidden Dim: 128
  • ODE Steps: 5
  • Solver Type: Euler
  • Sparsity Level: 0.7
  • Network Type: LTC
  [Training... 50 epochs]
  Result: Sharpe=1.42, Inference=95μs, Combined=1.45

Trial   2/30: [TPE sampling based on Trial 1...]
  [...]

[Progress bar: ============================> 50% (15/30 trials)]
ETA: 2:15:00 remaining
```

### Phase 3: Result Analysis (10 minutes)

```
Best Trial: 23/30
Best Hyperparameters:
  • Learning Rate: 0.001
  • Batch Size: 64
  • Hidden Dim: 256
  • ODE Steps: 10
  • Solver Type: RK4
  • Sparsity Level: 0.7
  • Network Type: CfC
  • Time Constant τ: 0.1
  • Adaptive τ: true

Performance:
  • Sharpe Ratio: 1.58
  • Inference Time: 87 μs
  • Combined Score: 1.61

Improvement over baseline: +12.4%
```

---

## Analysis Framework

### 1. ODE Integration Analysis

**Objective**: Find optimal balance between accuracy and speed

#### Metrics to Compare
- **Euler Solver**: Fastest, suitable for low-volatility regimes
- **RK4 Solver**: Most accurate, higher computational cost
- **Adaptive Solver**: Best of both worlds

#### Expected Results Table

| Solver   | ODE Steps | Sharpe Ratio | Inference (μs) | Combined Score |
|----------|-----------|--------------|----------------|----------------|
| Euler    | 3         | 1.35         | 45             | 1.45           |
| Euler    | 5         | 1.42         | 65             | 1.47           |
| RK4      | 5         | 1.48         | 110            | 1.45           |
| RK4      | 10        | 1.55         | 180            | 1.42           |
| Adaptive | 5         | 1.52         | 85             | 1.57           |

**Key Insight**: Adaptive solver likely provides best combined score by dynamically switching based on market conditions.

### 2. Sparsity Level Analysis

**Objective**: Determine optimal connection pruning for inference speed

#### Tradeoff Analysis

| Sparsity | Parameters | Memory (KB) | Inference (μs) | Sharpe Ratio | Combined Score |
|----------|------------|-------------|----------------|--------------|----------------|
| 0.5      | 65,536     | 256         | 120            | 1.58         | 1.55           |
| 0.7      | 32,768     | 128         | 85             | 1.52         | 1.57           |
| 0.9      | 8,192      | 32          | 55             | 1.38         | 1.47           |

**Key Insight**: 0.7 sparsity level likely optimal - balances accuracy (1.52 Sharpe) with speed (85 μs).

### 3. Continuous-Time Advantage Analysis

**Objective**: Validate superiority over discrete-time models (LSTM, DQN)

#### Model Comparison

| Model    | Architecture      | Inference (μs) | Sharpe Ratio | Training Time | Adaptability |
|----------|-------------------|----------------|--------------|---------------|--------------|
| LSTM     | Discrete recurrent| 500-1000       | 1.35         | Fast          | Low          |
| DQN      | Discrete RL       | 100-200        | 1.42         | Moderate      | Medium       |
| Liquid NN| Continuous ODE    | **<100**       | **>1.5**     | Slow          | **High**     |

**Key Advantages**:
1. **Speed**: Sparse connectivity → 2-10x faster than LSTM
2. **Accuracy**: ODE integration → smooth continuous dynamics
3. **Adaptability**: Time constant modulation → regime-aware behavior

### 4. Market Regime Adaptation

**Time Step Adjustment Based on Volatility**:
- **Normal**: dt = 0.01 (base time constant)
- **Sideways**: dt = 0.005 (slower adaptation)
- **Trending**: dt = 0.02 (faster adaptation)
- **Bull/Bear**: dt = 0.0025 (high-frequency updates)
- **Crisis**: dt = 0.00125 (ultra-fast response)

**Expected Impact**: 5-10% improvement in Sharpe ratio during volatile periods.

---

## Output Files

### Generated Artifacts

```
ml/trained_models/tuning/liquid_nn/
├── tuning_execution.log              # Full execution log
├── job_id.txt                         # Tuning job UUID
├── best_hyperparameters.txt          # Best trial results
├── LIQUID_NN_TUNING_SUMMARY.md       # Comprehensive report
├── checkpoints/                       # Model checkpoints per trial
│   ├── trial_01_epoch_50.safetensors
│   ├── trial_02_epoch_50.safetensors
│   └── ...
├── plots/                             # Visualization plots
│   ├── ode_integration_comparison.png
│   ├── sparsity_performance.png
│   ├── inference_time_distribution.png
│   └── sharpe_ratio_progression.png
└── analysis/                          # Detailed analyses
    ├── ode_integration_analysis.md
    ├── sparsity_analysis.md
    └── continuous_time_advantage.md
```

### Summary Report Structure

```markdown
# Liquid NN Tuning Summary

## Best Hyperparameters
[Full configuration]

## ODE Integration Analysis
- Solver comparison (Euler vs RK4 vs Adaptive)
- Accuracy vs speed tradeoff
- Optimal ODE steps

## Sparsity Analysis
- Connection pruning impact
- Inference speed vs accuracy
- Memory footprint

## Continuous-Time Advantage
- Comparison with LSTM/DQN
- Regime adaptation analysis
- HFT latency validation

## Production Recommendations
[Deployment configuration]
```

---

## Troubleshooting

### Common Issues

#### 1. GPU Memory Error
```
Error: CUDA out of memory
```
**Solution**: Reduce batch size in tuning_config.yaml or use CPU mode.

#### 2. gRPC Connection Failed
```
Error: Failed to connect to localhost:50054
```
**Solution**: Start ML Training Service:
```bash
cargo run -p ml_training_service --release &
```

#### 3. Missing Data Files
```
Warning: Missing data file: ES.FUT
```
**Solution**: Download DBN files or use available symbols only.

#### 4. Tuning Timeout
```
Error: Trial timeout after 60 minutes
```
**Solution**: Increase max_epochs_per_trial or reduce epochs in tuning_config.yaml.

---

## Next Steps

### After Tuning Completes

1. **Review Best Hyperparameters**
   ```bash
   cat ml/trained_models/tuning/liquid_nn/best_hyperparameters.txt
   ```

2. **Analyze ODE Integration**
   ```bash
   cat ml/trained_models/tuning/liquid_nn/analysis/ode_integration_analysis.md
   ```

3. **Validate Inference Speed**
   ```bash
   cargo run -p ml --example benchmark_liquid_nn --release
   ```

4. **Production Training**
   ```bash
   # Use best hyperparameters for full 100-epoch training
   cargo run -p ml_training_service --example train_liquid_nn_production
   ```

5. **Backtesting**
   ```bash
   # Test with best model on historical data
   cargo run -p backtesting_service --example test_liquid_nn_strategy
   ```

---

## Technical Deep Dive

### ODE Integration Mathematics

#### Euler Method (1st Order)
```
x(t + dt) = x(t) + dt × f(x(t), t)
```
- **Error**: O(dt)
- **Operations**: 1 function evaluation per step
- **Use Case**: Low volatility, speed-critical

#### RK4 Method (4th Order)
```
k1 = f(x(t), t)
k2 = f(x(t) + dt/2 × k1, t + dt/2)
k3 = f(x(t) + dt/2 × k2, t + dt/2)
k4 = f(x(t) + dt × k3, t + dt)
x(t + dt) = x(t) + dt/6 × (k1 + 2k2 + 2k3 + k4)
```
- **Error**: O(dt^4)
- **Operations**: 4 function evaluations per step
- **Use Case**: High accuracy required

#### Adaptive Method
```python
if market_regime in [Bull, Bear, Crisis]:
    solver = RK4  # High accuracy for volatile markets
else:
    solver = Euler  # Fast for normal conditions
```

### Sparsity Implementation

**Connection Pruning Strategy**:
```rust
// Prune weights below threshold
let threshold = percentile(weights, sparsity_level);
for weight in weights {
    if abs(weight) < threshold {
        weight = 0.0;
    }
}

// Inference with sparse matrix multiplication
output = sparse_matmul(sparse_weights, input);
```

**Memory Benefits**:
- 0.5 sparsity: 50% memory reduction
- 0.7 sparsity: 70% memory reduction
- 0.9 sparsity: 90% memory reduction

**Inference Benefits**:
- Fewer multiplications → faster compute
- Better cache locality → reduced memory bandwidth
- Suitable for fixed-point arithmetic

---

## Performance Expectations

### Baseline Targets

| Metric                  | Target    | Rationale                              |
|-------------------------|-----------|----------------------------------------|
| Sharpe Ratio            | >1.5      | Risk-adjusted returns superiority      |
| Inference Time          | <100 μs   | HFT latency requirement                |
| Combined Score          | >1.4      | sharpe - 0.1×log(inference_ms)         |
| Training Time/Trial     | 8-12 min  | 50 epochs × 10-15 sec/epoch            |
| Total Tuning Duration   | 4-6 hours | 30 trials × 8-12 min/trial             |
| GPU Memory Usage        | <3.5 GB   | RTX 3050 Ti 4GB constraint             |

### Success Criteria

✅ **Minimum Acceptable**:
- Sharpe Ratio: 1.3
- Inference Time: 150 μs
- Combined Score: 1.2

✅ **Target Performance**:
- Sharpe Ratio: 1.5
- Inference Time: 100 μs
- Combined Score: 1.4

✅ **Stretch Goal**:
- Sharpe Ratio: 1.8
- Inference Time: 80 μs
- Combined Score: 1.75

---

## References

### Papers & Research

1. **Liquid Time-constant Networks** (Hasani et al., 2021)
   - https://arxiv.org/abs/2006.04439
   - Introduces LTC neurons with adaptive time constants

2. **Neural Ordinary Differential Equations** (Chen et al., 2018)
   - https://arxiv.org/abs/1806.07366
   - Foundational work on continuous-time neural networks

3. **Closed-form Continuous-time Neural Networks** (Hasani et al., 2022)
   - https://www.nature.com/articles/s42256-022-00556-7
   - CfC neurons with closed-form solutions

### Foxhunt Documentation

- **CLAUDE.md**: System architecture and ML infrastructure
- **ML_TRAINING_ROADMAP.md**: 4-6 week training plan
- **GPU_TRAINING_BENCHMARK.md**: RTX 3050 Ti performance benchmarks
- **TESTING_PLAN.md**: ML model validation strategy

---

## Conclusion

This tuning campaign represents a significant milestone in Foxhunt's ML evolution:

1. **Innovation**: First continuous-time neural ODE model in HFT system
2. **Performance**: Ultra-low latency (<100μs) with high accuracy (Sharpe >1.5)
3. **Adaptability**: Market regime-aware behavior via time constant modulation
4. **Efficiency**: Sparse connectivity for faster inference and lower memory

**Expected Outcome**: Production-ready Liquid NN hyperparameters optimized for ODE integration, sparsity, and inference speed - ready for 100-epoch production training and live trading deployment.

---

**Status**: Ready to Execute
**Command**: `./run_liquid_nn_tuning.sh`
**Duration**: 4-6 hours
**Next Milestone**: Production training with optimized hyperparameters