foxhunt/docs/archive/historical/FAILED_TESTS_DEBUG_GUIDE.md

Failed Tests Debug Guide

Date: October 15, 2025 Total Failures: 9 tests

---

🔴 HIGH PRIORITY (3 tests - Production Critical)

1. Ensemble Decision Weight Adjustment

Test: ensemble::decision::tests::test_model_weight_adjustment File: /home/jgrusewski/Work/foxhunt/ml/src/ensemble/decision.rs Module: Ensemble voting and decision making Likely Cause: Weight normalization or Sharpe ratio calculation Impact: CRITICAL - Affects production ensemble predictions Debug Command:

cargo test -p ml ensemble::decision::tests::test_model_weight_adjustment -- --nocapture

2. DQN Feature-to-State Conversion

Test: trainers::dqn::tests::test_features_to_state File: /home/jgrusewski/Work/foxhunt/ml/src/trainers/dqn.rs Module: DQN feature engineering Likely Cause: Feature dimension mismatch (expected 256-dim state vector) Impact: CRITICAL - Breaks DQN training pipeline Debug Command:

cargo test -p ml trainers::dqn::tests::test_features_to_state -- --nocapture

3. DBN Data Loading Pipeline

Test: test_scenario_01_dbn_data_loading_pipeline File: /home/jgrusewski/Work/foxhunt/ml/tests/e2e_ensemble_integration.rs Module: End-to-end data pipeline integration Likely Cause: DBN file path or feature extraction issue Impact: CRITICAL - Prevents loading real market data Debug Command:

cargo test -p ml --test e2e_ensemble_integration test_scenario_01_dbn_data_loading_pipeline -- --nocapture

---

🟡 MEDIUM PRIORITY (3 tests)

4. Checkpoint Signer Model Types

Test: checkpoint::signer::tests::test_different_model_types File: /home/jgrusewski/Work/foxhunt/ml/src/checkpoint/signer.rs Module: Checkpoint signing and verification Likely Cause: Model type enum handling or signature mismatch Impact: MEDIUM - Affects checkpoint security Debug Command:

cargo test -p ml checkpoint::signer::tests::test_different_model_types -- --nocapture

5. Ensemble Performance Tracker

Test: ensemble::coordinator_extended::tests::test_performance_tracker File: /home/jgrusewski/Work/foxhunt/ml/src/ensemble/coordinator_extended.rs Module: Ensemble coordinator monitoring Likely Cause: Metrics collection or time-series data issue Impact: MEDIUM - Affects monitoring, not core predictions Debug Command:

cargo test -p ml ensemble::coordinator_extended::tests::test_performance_tracker -- --nocapture

6. Model Drift Detection

Test: security::anomaly_detector::tests::test_model_drift_detection File: /home/jgrusewski/Work/foxhunt/ml/src/security/anomaly_detector.rs Module: Security and anomaly detection Likely Cause: Drift threshold or statistical calculation Impact: MEDIUM - Affects monitoring, not core trading Debug Command:

cargo test -p ml security::anomaly_detector::tests::test_model_drift_detection -- --nocapture

---

🟢 LOW PRIORITY (3 tests - Benchmark Utilities)

7. Gradient Norm Calculation

Test: benchmark::stability_validator::tests::test_gradient_norm_calculation File: /home/jgrusewski/Work/foxhunt/ml/src/benchmark/stability_validator.rs Module: GPU training benchmark utilities Likely Cause: Unwrap panic on tensor operation or CUDA device access Impact: LOW - Benchmark utility, not production training Debug Command:

cargo test -p ml benchmark::stability_validator::tests::test_gradient_norm_calculation -- --nocapture

8. Outlier Detection

Test: benchmark::statistical_sampler::tests::test_outlier_detection File: /home/jgrusewski/Work/foxhunt/ml/src/benchmark/statistical_sampler.rs Module: Statistical sampling for benchmarks Likely Cause: Statistical threshold assertion failure Impact: LOW - Affects benchmark rigor, not training Debug Command:

cargo test -p ml benchmark::statistical_sampler::tests::test_outlier_detection -- --nocapture

9. Outlier Percentage

Test: benchmark::statistical_sampler::tests::test_outlier_percentage File: /home/jgrusewski/Work/foxhunt/ml/src/benchmark/statistical_sampler.rs Module: Statistical sampling for benchmarks Likely Cause: Related to test_outlier_detection (percentage calculation) Impact: LOW - Affects benchmark rigor, not training Debug Command:

cargo test -p ml benchmark::statistical_sampler::tests::test_outlier_percentage -- --nocapture

---

Common Debug Patterns

Check Feature Dimensions

// Expected DQN state size: 256 dimensions
// Check in: ml/src/trainers/dqn.rs
pub fn features_to_state(features: &[f64]) -> Result<Vec<f64>> {
    if features.len() != 256 {
        return Err(format!("Expected 256 features, got {}", features.len()));
    }
    // ...
}

Check DBN File Paths

// Test data location: /home/jgrusewski/Work/foxhunt/test_data/
// Verify files exist:
// - ES.FUT.dbn.zst (1,674 bars)
// - ZN.FUT.dbn.zst (28,935 bars)
// - 6E.FUT.dbn.zst (29,937 bars)

Check Ensemble Weight Normalization

// Weights should sum to 1.0
// Check in: ml/src/ensemble/decision.rs
let sum: f64 = weights.iter().sum();
let normalized: Vec<f64> = weights.iter().map(|w| w / sum).collect();

---

Batch Debug Commands

Run All Failed Tests

cargo test -p ml \
  ensemble::decision::tests::test_model_weight_adjustment \
  trainers::dqn::tests::test_features_to_state \
  checkpoint::signer::tests::test_different_model_types \
  ensemble::coordinator_extended::tests::test_performance_tracker \
  security::anomaly_detector::tests::test_model_drift_detection \
  benchmark::stability_validator::tests::test_gradient_norm_calculation \
  benchmark::statistical_sampler::tests::test_outlier_detection \
  benchmark::statistical_sampler::tests::test_outlier_percentage \
  -- --nocapture

cargo test -p ml --test e2e_ensemble_integration \
  test_scenario_01_dbn_data_loading_pipeline \
  -- --nocapture

Run High Priority Only

cargo test -p ml \
  ensemble::decision::tests::test_model_weight_adjustment \
  trainers::dqn::tests::test_features_to_state \
  -- --nocapture

cargo test -p ml --test e2e_ensemble_integration \
  test_scenario_01_dbn_data_loading_pipeline \
  -- --nocapture

---

Fix Verification

After fixing, verify with:

# Quick check (high priority only)
cargo test -p ml ensemble::decision trainers::dqn --lib -- --nocapture
cargo test -p ml --test e2e_ensemble_integration -- --nocapture

# Full ML crate check
cargo test -p ml --lib --skip cuda -- --nocapture

# Full integration check
cargo test -p ml --test e2e_ensemble_integration -- --nocapture

---

Success Criteria

High Priority Fixed

• ✅ test_model_weight_adjustment passes
• ✅ test_features_to_state passes
• ✅ test_scenario_01_dbn_data_loading_pipeline passes

Overall Target

• ✅ ML crate: >99% pass rate (770+/780 tests)
• ✅ Integration: 100% pass rate (13/13 tests)
• ✅ Workspace: >99% pass rate (1,220+/1,223 tests)

---

Last Updated: October 15, 2025 Next Review: After high-priority fixes