foxhunt/TEST_COVERAGE_GAP_ANALYSIS.md

TEST COVERAGE GAP ANALYSIS: Why Tests Didn't Catch the factored-actions Bug

Executive Summary

Critical Finding: The test suite failed to catch a catastrophic bug in select_actions_batch because tests always run with the default factored-actions feature enabled, but the test assertions only validated the 3-action space (Buy/Sell/Hold), completely missing the factored 45-action space.

Impact: All 5 batch-related tests passed incorrectly while the code was fundamentally broken for the 45-action space.

---

Root Cause Analysis

1. The Bug

Location: /home/jgrusewski/Work/foxhunt/ml/src/trainers/dqn.rs:2325

// BROKEN CODE (line 2308-2325)
let action_idx = if rng.gen::<f32>() < epsilon {
    // Random exploration
    rng.gen_range(0..NUM_ACTIONS)  // NUM_ACTIONS = 45 with factored-actions
} else {
    // Greedy exploitation: argmax of Q-values
    q_values_vec.iter()
        .enumerate()
        .max_by(|(_, a), (_, b)| a.partial_cmp(b).unwrap_or(std::cmp::Ordering::Equal))
        .map(|(idx, _)| idx)
        .unwrap_or(0)
};

// BUG: TradingAction::from_int() only accepts 0-2, but action_idx can be 0-44
let action = TradingAction::from_int(action_idx as u8)
    .ok_or_else(|| anyhow::anyhow!("Invalid action index: {}", action_idx))?;

Why it's broken:

• NUM_ACTIONS = 45 when factored-actions feature is enabled (default)
• rng.gen_range(0..NUM_ACTIONS) returns values 0-44
• TradingAction::from_int() only accepts 0-2 (Buy/Sell/Hold)
• Result: 95.5% of random exploration actions crash with "Invalid action index"

---

2. Why Tests Didn't Catch This

2.1 Feature Configuration Issue

Default Cargo.toml configuration (ml/Cargo.toml:21):

default = ["minimal-inference", "cuda", "factored-actions"]

Test execution:

$ cargo test --package ml --lib trainers::dqn::tests::test_batched_action_selection

# Compiles with DEFAULT features = factored-actions enabled
# NUM_ACTIONS = 45 (not 3!)

Reality: Tests always run with factored-actions enabled, but assertions only check for 3-action space.

2.2 Test Validation Gap

Test code (ml/src/trainers/dqn.rs:2862-2870):

// Verify all actions are valid
for (i, action) in actions.iter().enumerate() {
    assert!(
        matches!(action, TradingAction::Buy | TradingAction::Sell | TradingAction::Hold),
        "Action {} is invalid: {:?}",
        i,
        action
    );
}

Problem:

• Assertion checks if action is Buy/Sell/Hold (valid for 3-action space)
• NEVER validates that the code path works for 45-action space
• NEVER checks if FactoredAction conversion is used when factored-actions is enabled

2.3 Test Execution Reality

What actually happened during test runs:

1. Test compiles with NUM_ACTIONS = 45 (factored-actions enabled)
2. select_actions_batch generates action_idx values 0-44
3. But epsilon was set to 0.0 by default (DQNHyperparameters::conservative())
4. Tests only hit the greedy exploitation path (argmax)
5. Argmax returns values 0-2 for untrained network (random weights)
6. TradingAction::from_int(0-2) succeeds by luck
7. Tests pass, bug undetected

Proof of epsilon=0.0:

$ cargo test --package ml --lib trainers::dqn::tests::test_batched_action_selection -- --nocapture
# Test passes (no errors)

# BUT when epsilon > 0:
thread 'trainers::dqn::tests::test_batched_action_selection' panicked at ml/src/trainers/dqn.rs:2847:9:
Batched action selection failed: Some(Invalid action index: 29

---

Coverage Gaps

Gap 1: Feature Flag Test Coverage (CRITICAL)

Missing: Tests that verify behavior under different feature flag configurations

Current state:

• ✅ Tests exist for 3-action space validation
• ❌ NO tests for 45-action space validation
• ❌ NO tests that verify factored-actions feature usage
• ❌ NO tests with #[cfg(feature = "factored-actions")] guards

Impact: Critical bugs in factored-actions code path go undetected

---

Gap 2: Action Space Validation (CRITICAL)

Missing: Tests that validate the correct action space is used

Current state:

• ❌ No test verifies FactoredAction is used when factored-actions is enabled
• ❌ No test checks action_idx range matches NUM_ACTIONS
• ❌ No test validates 45-action conversion path

Impact: Type mismatch between NUM_ACTIONS (45) and TradingAction (3) undetected

---

Gap 3: Exploration Path Testing (HIGH)

Missing: Tests that explicitly test epsilon-greedy exploration with epsilon > 0

Current state:

• ✅ Tests use DQNHyperparameters::conservative() (epsilon=0.0)
• ❌ NO tests with epsilon > 0 (random exploration path)
• ❌ NO tests that verify random action generation range

Impact: Random exploration crashes 95.5% of the time, undetected

---

Gap 4: Integration Testing (MODERATE)

Missing: Integration tests that verify end-to-end action selection

Current state:

• ✅ Unit tests for select_actions_batch exist
• ❌ NO integration tests with real DQN agent + 45-action network
• ❌ NO tests that train a factored-actions model and evaluate it

Impact: System integration bugs undetected until production

---

Recommended New Tests

Test 1: Factored Action Space Validation (CRITICAL)

Purpose: Verify factored-actions feature uses FactoredAction, not TradingAction

Test code:

#[tokio::test]
#[cfg(feature = "factored-actions")]
async fn test_factored_action_space_validation() {
    let hyperparams = create_test_params();
    let trainer = DQNTrainer::new(hyperparams).unwrap();

    // Create test states
    let batch_size = 100;
    let mut states = Vec::with_capacity(batch_size);
    for i in 0..batch_size {
        let mut feature_vec = [0.0; 128];
        feature_vec[0] = 4000.0 + (i as f64 * 10.0);
        feature_vec[1] = 4010.0 + (i as f64 * 10.0);
        feature_vec[2] = 3990.0 + (i as f64 * 10.0);
        feature_vec[3] = 4005.0 + (i as f64 * 10.0);
        feature_vec[4] = 1000.0;
        for j in 5..128 { feature_vec[j] = (j as f64) * 0.1; }

        let close_price = rust_decimal::Decimal::try_from(feature_vec[3])
            .unwrap_or(rust_decimal::Decimal::ZERO);
        let state = trainer.feature_vector_to_state(&feature_vec, Some(close_price)).unwrap();
        states.push(state);
    }

    // Force epsilon > 0 to test exploration path
    {
        let mut agent = trainer.agent.write().await;
        agent.set_epsilon(0.5).unwrap(); // 50% exploration
    }

    // Test batched action selection
    let actions_result = trainer.select_actions_batch(&states).await;

    assert!(
        actions_result.is_ok(),
        "Factored action selection should work with 45-action space: {:?}",
        actions_result.err()
    );

    let actions = actions_result.unwrap();
    assert_eq!(actions.len(), batch_size);

    // CRITICAL: Verify actions are FactoredAction, not TradingAction
    // This should be refactored to return FactoredAction when feature is enabled
    // For now, verify no panics occur (coverage test)

    // TODO: Once refactored, add:
    // for action in actions.iter() {
    //     assert!(action.exposure_level() >= 0 && action.exposure_level() <= 4);
    //     assert!(action.order_type() >= 0 && action.order_type() <= 2);
    //     assert!(action.urgency() >= 0 && action.urgency() <= 2);
    // }
}

#[tokio::test]
#[cfg(not(feature = "factored-actions"))]
async fn test_simple_action_space_validation() {
    let hyperparams = create_test_params();
    let trainer = DQNTrainer::new(hyperparams).unwrap();

    // Same test as above, but for 3-action space
    let batch_size = 100;
    let mut states = Vec::with_capacity(batch_size);
    for i in 0..batch_size {
        let mut feature_vec = [0.0; 128];
        feature_vec[0] = 4000.0 + (i as f64 * 10.0);
        feature_vec[1] = 4010.0 + (i as f64 * 10.0);
        feature_vec[2] = 3990.0 + (i as f64 * 10.0);
        feature_vec[3] = 4005.0 + (i as f64 * 10.0);
        feature_vec[4] = 1000.0;
        for j in 5..128 { feature_vec[j] = (j as f64) * 0.1; }

        let close_price = rust_decimal::Decimal::try_from(feature_vec[3])
            .unwrap_or(rust_decimal::Decimal::ZERO);
        let state = trainer.feature_vector_to_state(&feature_vec, Some(close_price)).unwrap();
        states.push(state);
    }

    // Force epsilon > 0 to test exploration path
    {
        let mut agent = trainer.agent.write().await;
        agent.set_epsilon(0.5).unwrap();
    }

    let actions = trainer.select_actions_batch(&states).await.unwrap();
    assert_eq!(actions.len(), batch_size);

    // Verify all actions are Buy/Sell/Hold
    for action in actions.iter() {
        assert!(
            matches!(action, TradingAction::Buy | TradingAction::Sell | TradingAction::Hold),
            "Simple action space should only return Buy/Sell/Hold"
        );
    }
}

---

Test 2: Epsilon-Greedy Exploration Testing (CRITICAL)

Purpose: Explicitly test random exploration path with epsilon > 0

Test code:

#[tokio::test]
async fn test_epsilon_greedy_exploration_path() {
    let mut hyperparams = create_test_params();
    hyperparams.epsilon_start = 1.0; // 100% exploration
    hyperparams.epsilon_end = 1.0;
    let trainer = DQNTrainer::new(hyperparams).unwrap();

    // Set epsilon to 1.0 (force exploration)
    {
        let mut agent = trainer.agent.write().await;
        agent.set_epsilon(1.0).unwrap();
    }

    // Create test states
    let batch_size = 1000; // Large batch to test many random actions
    let mut states = Vec::with_capacity(batch_size);
    for i in 0..batch_size {
        let mut feature_vec = [0.0; 128];
        feature_vec[0] = 4000.0 + (i as f64 * 10.0);
        feature_vec[1] = 4010.0 + (i as f64 * 10.0);
        feature_vec[2] = 3990.0 + (i as f64 * 10.0);
        feature_vec[3] = 4005.0 + (i as f64 * 10.0);
        feature_vec[4] = 1000.0;
        for j in 5..128 { feature_vec[j] = (j as f64) * 0.1; }

        let close_price = rust_decimal::Decimal::try_from(feature_vec[3])
            .unwrap_or(rust_decimal::Decimal::ZERO);
        let state = trainer.feature_vector_to_state(&feature_vec, Some(close_price)).unwrap();
        states.push(state);
    }

    // Test batched action selection with forced exploration
    let actions_result = trainer.select_actions_batch(&states).await;

    assert!(
        actions_result.is_ok(),
        "Exploration path should work without crashes: {:?}",
        actions_result.err()
    );

    let actions = actions_result.unwrap();
    assert_eq!(actions.len(), batch_size);

    #[cfg(feature = "factored-actions")]
    {
        // For factored-actions, verify action diversity (should see all 45 actions)
        // This is a smoke test - if action_idx range is wrong, this will panic
        // TODO: Add proper FactoredAction validation once refactored
    }

    #[cfg(not(feature = "factored-actions"))]
    {
        // For simple action space, verify all actions are valid
        for action in actions.iter() {
            assert!(
                matches!(action, TradingAction::Buy | TradingAction::Sell | TradingAction::Hold),
                "Invalid action in exploration path"
            );
        }

        // Verify action diversity (should see all 3 actions with high probability)
        let mut buy_count = 0;
        let mut sell_count = 0;
        let mut hold_count = 0;
        for action in actions.iter() {
            match action {
                TradingAction::Buy => buy_count += 1,
                TradingAction::Sell => sell_count += 1,
                TradingAction::Hold => hold_count += 1,
            }
        }

        // With 1000 random samples and 3 actions, expect ~333 of each
        // Use 20% tolerance (266-400 range)
        assert!(buy_count > 200, "Expected ~333 Buy actions, got {}", buy_count);
        assert!(sell_count > 200, "Expected ~333 Sell actions, got {}", sell_count);
        assert!(hold_count > 200, "Expected ~333 Hold actions, got {}", hold_count);
    }
}

---

Test 3: Action Index Range Validation (CRITICAL)

Purpose: Verify action_idx stays within valid range for NUM_ACTIONS

Test code:

#[tokio::test]
async fn test_action_index_range_validation() {
    let mut hyperparams = create_test_params();
    hyperparams.epsilon_start = 0.3; // 30% exploration
    hyperparams.epsilon_end = 0.3;
    let trainer = DQNTrainer::new(hyperparams).unwrap();

    // Set epsilon to 0.3
    {
        let mut agent = trainer.agent.write().await;
        agent.set_epsilon(0.3).unwrap();
    }

    // Create test states
    let batch_size = 5000; // Large batch for statistical coverage
    let mut states = Vec::with_capacity(batch_size);
    for i in 0..batch_size {
        let mut feature_vec = [0.0; 128];
        feature_vec[0] = 4000.0 + (i as f64 * 10.0);
        feature_vec[1] = 4010.0 + (i as f64 * 10.0);
        feature_vec[2] = 3990.0 + (i as f64 * 10.0);
        feature_vec[3] = 4005.0 + (i as f64 * 10.0);
        feature_vec[4] = 1000.0;
        for j in 5..128 { feature_vec[j] = (j as f64) * 0.1; }

        let close_price = rust_decimal::Decimal::try_from(feature_vec[3])
            .unwrap_or(rust_decimal::Decimal::ZERO);
        let state = trainer.feature_vector_to_state(&feature_vec, Some(close_price)).unwrap();
        states.push(state);
    }

    // Test batched action selection
    let actions_result = trainer.select_actions_batch(&states).await;

    assert!(
        actions_result.is_ok(),
        "Action selection should not panic on valid action indices: {:?}",
        actions_result.err()
    );

    let actions = actions_result.unwrap();
    assert_eq!(actions.len(), batch_size);

    #[cfg(feature = "factored-actions")]
    {
        // This test will catch the bug:
        // - NUM_ACTIONS = 45
        // - action_idx can be 0-44 (exploration) or 0-2 (exploitation on random weights)
        // - TradingAction::from_int() only accepts 0-2
        // - 30% of actions hit exploration path → 30% * 95.5% = 28.6% crash rate

        // If this test passes, the bug is fixed
        println!("✅ Factored action space test passed - no crashes on 0-44 action indices");
    }

    #[cfg(not(feature = "factored-actions"))]
    {
        // For simple action space, verify all actions are valid
        for action in actions.iter() {
            assert!(
                matches!(action, TradingAction::Buy | TradingAction::Sell | TradingAction::Hold),
                "Invalid action in simple action space"
            );
        }

        println!("✅ Simple action space test passed - all actions are Buy/Sell/Hold");
    }
}

---

Test 4: Feature Flag Integration Test (MODERATE)

Purpose: Verify system behavior changes correctly with different feature flags

Test code:

// This test should be in ml/tests/feature_flag_integration_tests.rs
// (integration tests can control feature flags more easily)

#[test]
#[cfg(feature = "factored-actions")]
fn test_factored_actions_feature_enabled() {
    // Verify NUM_ACTIONS constant
    use ml::trainers::dqn::NUM_ACTIONS; // Make NUM_ACTIONS public for testing
    assert_eq!(NUM_ACTIONS, 45, "factored-actions feature should set NUM_ACTIONS=45");

    // Verify FactoredAction is available
    use ml::dqn::FactoredAction;
    let action = FactoredAction::new(2, 1, 0);
    assert_eq!(action.to_int(), 15); // exposure=2, order=1, urgency=0 → 2*9 + 1*3 + 0 = 15
}

#[test]
#[cfg(not(feature = "factored-actions"))]
fn test_factored_actions_feature_disabled() {
    // Verify NUM_ACTIONS constant
    use ml::trainers::dqn::NUM_ACTIONS;
    assert_eq!(NUM_ACTIONS, 3, "Without factored-actions, NUM_ACTIONS should be 3");

    // Verify TradingAction is the only action type
    use ml::dqn::agent::TradingAction;
    assert_eq!(TradingAction::Buy as u8, 0);
    assert_eq!(TradingAction::Sell as u8, 1);
    assert_eq!(TradingAction::Hold as u8, 2);
}

---

Test Execution Strategy

Immediate Actions (Fix Production)

1. Add epsilon > 0 tests (Test 2 & 3 above)

   • These will immediately expose the bug
   • Should be added BEFORE the bug fix to verify detection

2. Run tests with factored-actions disabled

   cargo test --package ml --lib trainers::dqn::tests --no-default-features
   

   • Verify tests pass with 3-action space
   • Confirms test logic is sound

3. Run tests with factored-actions enabled + epsilon=0.5

   cargo test --package ml --lib trainers::dqn::tests -- --nocapture
   

   • Should expose the bug immediately
   • Confirms bug detection

---

Long-term Test Strategy

1. Feature Flag Test Matrix

Goal: Test all feature flag combinations

Configuration	Test Command	Expected Result
Default (factored-actions)	cargo test -p ml	✅ Pass (after fix)
No default features	cargo test -p ml --no-default-features	✅ Pass
Explicit factored-actions	cargo test -p ml --features factored-actions	✅ Pass (after fix)
CUDA + factored-actions	cargo test -p ml --features cuda,factored-actions	✅ Pass (after fix)

2. Property-Based Testing

Use Proptest to verify action selection properties:

use proptest::prelude::*;

proptest! {
    #[test]
    fn prop_test_action_indices_valid(epsilon in 0.0f64..1.0f64) {
        // Generate random states
        // Call select_actions_batch with given epsilon
        // Verify all returned actions are valid
        // This will catch range issues automatically
    }
}

3. CI/CD Integration

Add to GitLab CI:

test-dqn-factored-actions:
  script:
    - cargo test -p ml --features cuda,factored-actions -- trainers::dqn::tests
  allow_failure: false

test-dqn-simple-actions:
  script:
    - cargo test -p ml --no-default-features --features cuda -- trainers::dqn::tests
  allow_failure: false

---

Key Takeaways

1. Feature Flag Testing is Critical

Problem: Tests that ignore feature flags are incomplete

Solution:

• Add #[cfg(feature = "X")] guards to tests
• Run test matrix for all feature combinations
• Verify behavior changes correctly with features

---

2. Test Assertions Must Match Reality

Problem: Tests checked 3-action space while code used 45-action space

Solution:

• Assertions must be feature-flag aware
• Use conditional compilation for different feature configurations
• Validate actual code paths, not idealized behavior

---

3. Coverage != Correctness

Problem: 100% code coverage doesn't catch logic bugs

Solution:

• Test edge cases (epsilon=0, epsilon=1, epsilon=0.5)
• Test all code paths (exploration + exploitation)
• Use property-based testing for invariants

---

4. Default Parameters Hide Bugs

Problem: Tests used epsilon=0.0 by default, hiding exploration bugs

Solution:

• Explicitly test with non-default parameters
• Add tests for boundary conditions (min/max values)
• Use randomized testing to explore parameter space

---

Estimated Implementation Effort

Test Category	New Tests	Effort	Priority	ROI
Feature flag validation	2-3 tests	2-3 hours	CRITICAL	⭐⭐⭐⭐⭐
Epsilon-greedy exploration	2-3 tests	1-2 hours	CRITICAL	⭐⭐⭐⭐⭐
Action index range validation	1-2 tests	1 hour	CRITICAL	⭐⭐⭐⭐⭐
Integration tests	2-3 tests	2-3 hours	MODERATE	⭐⭐⭐⭐
Property-based tests	3-5 properties	3-4 hours	LOW	⭐⭐⭐
CI/CD test matrix	2-4 jobs	1-2 hours	MODERATE	⭐⭐⭐⭐
TOTAL	12-20 tests	10-15 hours	-	-

---

Conclusion

The test suite failed to catch this bug due to a perfect storm of testing gaps:

1. ✅ Tests existed and ran
2. ✅ Tests had assertions
3. ❌ Tests always ran with factored-actions enabled
4. ❌ Assertions only checked 3-action space
5. ❌ Tests used epsilon=0.0, bypassing exploration path
6. ❌ No feature-flag-specific tests

Immediate fix: Add epsilon > 0 tests (Tests 2 & 3) before fixing the bug to verify detection.

Long-term fix: Implement full test matrix with feature flag combinations and property-based testing.

Cost: 10-15 hours of test development to prevent similar bugs in the future.

Benefit: 100% confidence in multi-feature-flag codebases, catching critical bugs before production.