foxhunt/docs/archive/agents/AGENT_86_QUICKSTART.md

Agent 86: Quickstart Guide - Execute Backtests

Prerequisites from Agent 85: Backtesting infrastructure complete, awaiting execution

---

Step 1: Check Cargo Lock Status (1 minute)

# Check if cargo processes are still running
ps aux | grep cargo | grep -v grep

# If processes are running, wait or kill them:
# Option A: Wait 5-10 minutes for natural completion
# Option B: Kill safe processes (NOT training jobs)

---

Step 2: Verify Model Checkpoints (1 minute)

# Confirm PPO checkpoint exists
ls -lh ml/trained_models/production/ppo_real_data/ppo_checkpoint_epoch_500.safetensors

# Expected: 234 bytes (combined checkpoint file)
# Also check: ppo_actor_epoch_500.safetensors (42KB)
#             ppo_critic_epoch_500.safetensors (42KB)

---

Step 3: Build Backtest Script (2-5 minutes)

# Build in release mode for performance
cargo build -p ml --example comprehensive_model_backtest --release

# Expected output: Successful compilation
# If blocked: Wait for file lock to clear

---

Step 4: Execute Backtests (20-30 minutes)

# Run comprehensive backtest for available models (PPO + TLOB)
cargo run -p ml --example comprehensive_model_backtest --release

# Expected output:
# - Console progress for PPO and TLOB testing
# - Performance metrics (Sharpe, win rate, drawdown)
# - JSON results file: results/backtest_results_<timestamp>.json

---

Step 5: Verify Results (5 minutes)

# Check results directory
ls -lh results/

# View latest results
cat results/backtest_results_*.json | jq '.'

# Expected metrics (PPO):
# - Sharpe Ratio: >1.0 (target: >1.5)
# - Win Rate: >50% (target: >55%)
# - Max Drawdown: <20% (target: <15%)

---

Success Criteria

✅ PPO backtest executed without runtime errors ✅ TLOB backtest executed with fallback engine ✅ JSON results generated with performance metrics ✅ Sharpe ratio >1.0 for at least one model ✅ Win rate >50% for at least one model

---

If Backtests Fail

Scenario 1: Model Loading Error

Symptom: "Failed to load model" error Fix: Check checkpoint path and file permissions

ls -l ml/trained_models/production/ppo_real_data/*.safetensors
chmod 644 ml/trained_models/production/ppo_real_data/*.safetensors

Scenario 2: Data Loading Error

Symptom: "No DBN files found" error Fix: Verify test data directory

ls -lh test_data/real/databento/ml_training_small/
# Expected: ES.FUT, NQ.FUT, ZN.FUT, 6E.FUT DBN files

Scenario 3: Performance Below Targets

Symptom: Sharpe <1.0, win rate <50% Action: Document results and recommend hyperparameter tuning Note: Models may need optimization, not a failure condition

---

Expected Timeline

Step	Duration	Cumulative
Cargo lock check	1 min	1 min
Verify checkpoints	1 min	2 min
Build script	5 min	7 min
Execute backtests	30 min	37 min
Verify results	5 min	42 min
TOTAL	42 min	-

---

Deliverables

1. ✅ Backtest execution logs: Console output with progress
2. ✅ JSON results file: results/backtest_results_<timestamp>.json
3. ✅ Performance summary: Sharpe, win rate, drawdown for each model
4. ✅ Status report: Document which models passed/failed performance targets

---

Next Steps After Successful Execution

If Performance Meets Targets (Sharpe >1.5, Win Rate >55%)

→ Agent 87: Coordinate MAMBA-2 and TFT training, then full suite backtest

If Performance Below Targets (Sharpe <1.5, Win Rate <50%)

→ Hyperparameter Tuning: Use Optuna to optimize model parameters → Data Analysis: Check for data quality issues or market regime changes

If Models Missing (MAMBA-2, TFT, DQN)

→ ML Training Team: Re-train missing models with checkpoint verification → Timeline: 8-13 hours for complete model suite

---

Quick Command Reference

# Build backtest
cargo build -p ml --example comprehensive_model_backtest --release

# Run backtest
cargo run -p ml --example comprehensive_model_backtest --release

# View results
cat results/backtest_results_*.json | jq '.[] | {model: .model_name, sharpe: .sharpe_ratio, win_rate: .win_rate, pnl: .total_pnl}'

# Check model files
find ml/trained_models/production -name "*.safetensors" -size +10k -ls

# Verify data
ls -lh test_data/real/databento/ml_training_small/*.dbn

---

Created: 2025-10-14 by Agent 85 For: Agent 86 (Execute Available Backtests) Estimated Time: 42 minutes Success Rate: 95% (assuming cargo lock clears)