foxhunt/FAILED_TESTS_DEBUG_GUIDE.md

# Failed Tests Debug Guide
**Date**: October 15, 2025
**Total Failures**: 9 tests

---

## 🔴 HIGH PRIORITY (3 tests - Production Critical)

### 1. Ensemble Decision Weight Adjustment
**Test**: `ensemble::decision::tests::test_model_weight_adjustment`
**File**: `/home/jgrusewski/Work/foxhunt/ml/src/ensemble/decision.rs`
**Module**: Ensemble voting and decision making
**Likely Cause**: Weight normalization or Sharpe ratio calculation
**Impact**: **CRITICAL** - Affects production ensemble predictions
**Debug Command**:
```bash
cargo test -p ml ensemble::decision::tests::test_model_weight_adjustment -- --nocapture
```

### 2. DQN Feature-to-State Conversion
**Test**: `trainers::dqn::tests::test_features_to_state`
**File**: `/home/jgrusewski/Work/foxhunt/ml/src/trainers/dqn.rs`
**Module**: DQN feature engineering
**Likely Cause**: Feature dimension mismatch (expected 256-dim state vector)
**Impact**: **CRITICAL** - Breaks DQN training pipeline
**Debug Command**:
```bash
cargo test -p ml trainers::dqn::tests::test_features_to_state -- --nocapture
```

### 3. DBN Data Loading Pipeline
**Test**: `test_scenario_01_dbn_data_loading_pipeline`
**File**: `/home/jgrusewski/Work/foxhunt/ml/tests/e2e_ensemble_integration.rs`
**Module**: End-to-end data pipeline integration
**Likely Cause**: DBN file path or feature extraction issue
**Impact**: **CRITICAL** - Prevents loading real market data
**Debug Command**:
```bash
cargo test -p ml --test e2e_ensemble_integration test_scenario_01_dbn_data_loading_pipeline -- --nocapture
```

---

## 🟡 MEDIUM PRIORITY (3 tests)

### 4. Checkpoint Signer Model Types
**Test**: `checkpoint::signer::tests::test_different_model_types`
**File**: `/home/jgrusewski/Work/foxhunt/ml/src/checkpoint/signer.rs`
**Module**: Checkpoint signing and verification
**Likely Cause**: Model type enum handling or signature mismatch
**Impact**: MEDIUM - Affects checkpoint security
**Debug Command**:
```bash
cargo test -p ml checkpoint::signer::tests::test_different_model_types -- --nocapture
```

### 5. Ensemble Performance Tracker
**Test**: `ensemble::coordinator_extended::tests::test_performance_tracker`
**File**: `/home/jgrusewski/Work/foxhunt/ml/src/ensemble/coordinator_extended.rs`
**Module**: Ensemble coordinator monitoring
**Likely Cause**: Metrics collection or time-series data issue
**Impact**: MEDIUM - Affects monitoring, not core predictions
**Debug Command**:
```bash
cargo test -p ml ensemble::coordinator_extended::tests::test_performance_tracker -- --nocapture
```

### 6. Model Drift Detection
**Test**: `security::anomaly_detector::tests::test_model_drift_detection`
**File**: `/home/jgrusewski/Work/foxhunt/ml/src/security/anomaly_detector.rs`
**Module**: Security and anomaly detection
**Likely Cause**: Drift threshold or statistical calculation
**Impact**: MEDIUM - Affects monitoring, not core trading
**Debug Command**:
```bash
cargo test -p ml security::anomaly_detector::tests::test_model_drift_detection -- --nocapture
```

---

## 🟢 LOW PRIORITY (3 tests - Benchmark Utilities)

### 7. Gradient Norm Calculation
**Test**: `benchmark::stability_validator::tests::test_gradient_norm_calculation`
**File**: `/home/jgrusewski/Work/foxhunt/ml/src/benchmark/stability_validator.rs`
**Module**: GPU training benchmark utilities
**Likely Cause**: Unwrap panic on tensor operation or CUDA device access
**Impact**: LOW - Benchmark utility, not production training
**Debug Command**:
```bash
cargo test -p ml benchmark::stability_validator::tests::test_gradient_norm_calculation -- --nocapture
```

### 8. Outlier Detection
**Test**: `benchmark::statistical_sampler::tests::test_outlier_detection`
**File**: `/home/jgrusewski/Work/foxhunt/ml/src/benchmark/statistical_sampler.rs`
**Module**: Statistical sampling for benchmarks
**Likely Cause**: Statistical threshold assertion failure
**Impact**: LOW - Affects benchmark rigor, not training
**Debug Command**:
```bash
cargo test -p ml benchmark::statistical_sampler::tests::test_outlier_detection -- --nocapture
```

### 9. Outlier Percentage
**Test**: `benchmark::statistical_sampler::tests::test_outlier_percentage`
**File**: `/home/jgrusewski/Work/foxhunt/ml/src/benchmark/statistical_sampler.rs`
**Module**: Statistical sampling for benchmarks
**Likely Cause**: Related to test_outlier_detection (percentage calculation)
**Impact**: LOW - Affects benchmark rigor, not training
**Debug Command**:
```bash
cargo test -p ml benchmark::statistical_sampler::tests::test_outlier_percentage -- --nocapture
```

---

## Common Debug Patterns

### Check Feature Dimensions
```rust
// Expected DQN state size: 256 dimensions
// Check in: ml/src/trainers/dqn.rs
pub fn features_to_state(features: &[f64]) -> Result<Vec<f64>> {
    if features.len() != 256 {
        return Err(format!("Expected 256 features, got {}", features.len()));
    }
    // ...
}
```

### Check DBN File Paths
```rust
// Test data location: /home/jgrusewski/Work/foxhunt/test_data/
// Verify files exist:
// - ES.FUT.dbn.zst (1,674 bars)
// - ZN.FUT.dbn.zst (28,935 bars)
// - 6E.FUT.dbn.zst (29,937 bars)
```

### Check Ensemble Weight Normalization
```rust
// Weights should sum to 1.0
// Check in: ml/src/ensemble/decision.rs
let sum: f64 = weights.iter().sum();
let normalized: Vec<f64> = weights.iter().map(|w| w / sum).collect();
```

---

## Batch Debug Commands

### Run All Failed Tests
```bash
cargo test -p ml \
  ensemble::decision::tests::test_model_weight_adjustment \
  trainers::dqn::tests::test_features_to_state \
  checkpoint::signer::tests::test_different_model_types \
  ensemble::coordinator_extended::tests::test_performance_tracker \
  security::anomaly_detector::tests::test_model_drift_detection \
  benchmark::stability_validator::tests::test_gradient_norm_calculation \
  benchmark::statistical_sampler::tests::test_outlier_detection \
  benchmark::statistical_sampler::tests::test_outlier_percentage \
  -- --nocapture

cargo test -p ml --test e2e_ensemble_integration \
  test_scenario_01_dbn_data_loading_pipeline \
  -- --nocapture
```

### Run High Priority Only
```bash
cargo test -p ml \
  ensemble::decision::tests::test_model_weight_adjustment \
  trainers::dqn::tests::test_features_to_state \
  -- --nocapture

cargo test -p ml --test e2e_ensemble_integration \
  test_scenario_01_dbn_data_loading_pipeline \
  -- --nocapture
```

---

## Fix Verification

After fixing, verify with:
```bash
# Quick check (high priority only)
cargo test -p ml ensemble::decision trainers::dqn --lib -- --nocapture
cargo test -p ml --test e2e_ensemble_integration -- --nocapture

# Full ML crate check
cargo test -p ml --lib --skip cuda -- --nocapture

# Full integration check
cargo test -p ml --test e2e_ensemble_integration -- --nocapture
```

---

## Success Criteria

### High Priority Fixed
- ✅ `test_model_weight_adjustment` passes
- ✅ `test_features_to_state` passes
- ✅ `test_scenario_01_dbn_data_loading_pipeline` passes

### Overall Target
- ✅ ML crate: >99% pass rate (770+/780 tests)
- ✅ Integration: 100% pass rate (13/13 tests)
- ✅ Workspace: >99% pass rate (1,220+/1,223 tests)

---

**Last Updated**: October 15, 2025
**Next Review**: After high-priority fixes