foxhunt/LEGACY_256_TEST_CLEANUP.md

LEGACY TEST CODE AUDIT: 256-Feature References

Generated: 2025-10-20 Agent: Legacy Test Code Audit Context: Wave D complete with 225 features (201 Wave C + 24 Wave D). Legacy 256-feature test code documented for future cleanup.

---

EXECUTIVE SUMMARY

Files found: 45
Total legacy test lines: 1,209 (core) + ~3,500 (documentation)
Total 256 references: 140 (feature/dimension context)
Cleanup effort: 8-12 hours
Priority: MEDIUM

Status: Wave D production deployment NOT blocked. Legacy tests need updating before ML retraining (4-6 week roadmap item).

---

CATEGORY BREAKDOWN

CATEGORY 1: CRITICAL - FEATURE EXTRACTION TESTS ⚠️

3 files, 1,209 lines - MUST UPDATE

These explicitly test 256-dimensional feature extraction and will FAIL with current 225-feature implementation:

1. ml/tests/test_extract_256_dim_features.rs (206 lines)

// Current (BROKEN):
assert_eq!(feature_vec.len(), 256);

// Required:
assert_eq!(feature_vec.len(), 225);

• Impact: HIGH - Validates core feature extraction pipeline
• Update effort: 2 hours
• Changes: Assertions 256→225, test names, comments

2. ml/tests/dbn_256_feature_validation.rs (616 lines)

// Current (BROKEN):
for feat_idx in 0..256 {
    assert_eq!(report.feature_stats.len(), 256);
}

// Required:
for feat_idx in 0..225 {
    assert_eq!(report.feature_stats.len(), 225);
}

• Impact: HIGH - Real data validation (ES.FUT, NQ.FUT, 6E.FUT, ZN.FUT)
• Update effort: 3 hours
• Changes: Loop bounds, assertions, statistical analysis

3. ml/tests/test_dbn_sequence_256_features.rs (387 lines)

// Current (PARTIALLY BROKEN):
let mut loader = DbnSequenceLoader::with_limits(60, 256, Some(10), 10);

// Required (backward compatible):
let mut loader = DbnSequenceLoader::with_limits(60, 225, Some(10), 10);
// Also test: d_model in [128, 225, 256, 512]

• Impact: MEDIUM - Loader configuration test
• Update effort: 2 hours
• Changes: Default d_model=225, add multi-value tests

---

CATEGORY 2: MODEL CONFIGURATION TESTS ✅

18 files - NO CHANGES REQUIRED

These test model architectures where d_model=256 is a VALID configuration choice. Models support variable dimensions (128, 256, 512, etc.). DO NOT CHANGE.

Files:

1. ml/tests/e2e_mamba2_training.rs
2. ml/tests/mamba_training_test.rs
3. ml/tests/mamba2_e2e_training.rs
4. ml/tests/mamba2_training_pipeline_test.rs
5. ml/tests/ensemble_4_model_trainable_integration.rs
6. ml/tests/tft_quantized_attention_unit_test.rs
7. ml/tests/streaming_pipeline_edge_cases.rs
8. ml/tests/ppo_e2e_training.rs
9. ml/tests/liquid_networks_test.rs
10. ml/tests/gpu_4_model_stress_test.rs
11. ml/tests/gpu_memory_budget_validation.rs
12. ml/tests/memory_optimization_tests.rs
13. ml/tests/test_dbn_parser_fix.rs
14. ml/tests/test_streaming_loader.rs
15. ml/tests/varmap_weight_extraction_test.rs
16. ml/tests/checkpoint_test.rs
17. ml/tests/model_registry_checkpoint_test.rs
18. ml/tests/tft_int8_calibration_dataset_test.rs

Rationale: These tests validate model flexibility across different d_model values. The models MUST support 256 for backward compatibility and production flexibility.

---

CATEGORY 3: DOCUMENTATION/COMMENTS 📝

24 files, ~140 references - SHOULD UPDATE

Files with "256" in comments/docs referring to old feature count. Low priority, easy fix.

Update Strategy: Global search-replace in comments

# Example:
# "256-dim features" → "225-dim features"
# "256 features per bar" → "225 features per bar"
# "256-feature pipeline" → "225-feature pipeline"

Files:

1. ml/tests/feature_cache_tests.rs (comments: "256-dim vectors")
2. ml/tests/microstructure_tests.rs (comment: "256-feature pipeline")
3. ml/tests/alternative_bars_integration_test.rs (comment: "256 features per bar")
4. ml/tests/calibration_dataset_test.rs (comment: "256 features if using full MAMBA-2")
5. ml/tests/dbn_feature_config_test.rs
6. ml/tests/e2e_ensemble_integration.rs
7. ml/tests/model_registry_tests.rs
8. ml/tests/model_validation_comprehensive.rs
9. ml/tests/ppo_gae_test.rs
10. ml/tests/ppo_training_pipeline_test.rs
11. ml/tests/test_feature_cache_service.rs
12. ml/tests/test_tft_cuda_layernorm.rs
13. ml/tests/tft_attention_gradient_flow.rs
14. ml/tests/tft_attention_int8_quantization_test.rs
15. ml/tests/tft_e2e_training.rs
16. ml/tests/tft_inference_latency_benchmark.rs
17. ml/tests/tft_int8_inference_integration_test.rs
18. ml/tests/tft_int8_memory_benchmark_test.rs
19. ml/tests/tft_varmap_checkpoint_test.rs
20. ml/tests/training_chaos_tests.rs
21. ml/tests/wave_d_normalization_integration_test.rs
22. ml/tests/mamba_test.rs
23. ml/tests/mamba2_hardware_aware_test.rs
24. ml/tests/ensemble_4_model_trainable_integration.rs

Update effort: 1 hour (automated search-replace)

---

CATEGORY 4: NON-FEATURE USES ✅

Files using "256" for non-feature purposes - NO CHANGES

• ml/tests/unsafe_validation_tests.rs (buffer.set_len(256) - buffer capacity)
• ml/tests/liquid_networks_test.rs ((128, 256) - neural network layer sizes)
• ml/tests/mamba_training_test.rs (Tensor::randn(&[1, 128, 256]) - shape tests)

Rationale: Not related to feature dimensions, DO NOT CHANGE.

---

CLEANUP TASK LIST

✅ Phase 1: Critical Feature Tests (7 hours)

Task 1.1: Update test_extract_256_dim_features.rs (2h)

# File: ml/tests/test_extract_256_dim_features.rs
# Lines: 206
# Changes:
- [ ] Line 40-48: Change assertions 256 → 225
- [ ] Line 9: Rename test to test_extract_225_dim_features
- [ ] Line 1-4: Update file header documentation
- [ ] Verify with: cargo test --test test_extract_256_dim_features

Key Changes:

// Before:
assert_eq!(feature_vec.len(), 256, "Feature vector {} has wrong dimension", i);

// After:
assert_eq!(feature_vec.len(), 225, "Feature vector {} has wrong dimension", i);

Task 1.2: Update dbn_256_feature_validation.rs (3h)

# File: ml/tests/dbn_256_feature_validation.rs
# Lines: 616
# Changes:
- [ ] Line 1-21: Update file header (256 → 225)
- [ ] Line 172: Change loop bounds (0..256 → 0..225)
- [ ] Line 230, 242, 268, 294: Update assertions (256 → 225)
- [ ] Line 313, 331, 359: Update feature count assertions
- [ ] Line 411: Update performance targets
- [ ] Verify with: cargo test --test dbn_256_feature_validation

Key Changes:

// Before:
let mut feature_stats = Vec::with_capacity(256);
for feat_idx in 0..256 { /* ... */ }
assert_eq!(report.feature_stats.len(), 256);

// After:
let mut feature_stats = Vec::with_capacity(225);
for feat_idx in 0..225 { /* ... */ }
assert_eq!(report.feature_stats.len(), 225);

Task 1.3: Update test_dbn_sequence_256_features.rs (2h)

# File: ml/tests/test_dbn_sequence_256_features.rs
# Lines: 387
# Changes:
- [ ] Line 1-4: Update file header (256 → 225)
- [ ] Line 38: Change default d_model (256 → 225)
- [ ] Line 69, 98: Update shape documentation
- [ ] Line 237-259: Add multi-d_model test (128, 225, 256, 512)
- [ ] Verify backward compatibility
- [ ] Verify with: cargo test --test test_dbn_sequence_256_features

Key Changes:

// Before:
let mut loader = DbnSequenceLoader::with_limits(60, 256, Some(10), 10);
println!("✅ Created DbnSequenceLoader (seq_len=60, d_model=256, max=10, stride=10)\n");

// After (with backward compatibility test):
let mut loader = DbnSequenceLoader::with_limits(60, 225, Some(10), 10);
println!("✅ Created DbnSequenceLoader (seq_len=60, d_model=225, max=10, stride=10)\n");

// Add new test:
#[tokio::test]
async fn test_multiple_d_model_values() -> Result<()> {
    for d_model in [128, 225, 256, 512] {
        let mut loader = DbnSequenceLoader::with_limits(60, d_model, Some(10), 10).await?;
        // ... validate each d_model ...
    }
}

---

✅ Phase 2: Documentation Update (1 hour)

Task 2.1: Global Comment Update

# Search patterns:
grep -r "256.*feature" ml/tests/ --include="*.rs"
grep -r "256.*dim" ml/tests/ --include="*.rs"
grep -r "256-dimensional" ml/tests/ --include="*.rs"

# Replace (manual review required):
# "256-dim features" → "225-dim features"
# "256 features per bar" → "225 features per bar"
# "256-feature pipeline" → "225-feature pipeline"
# "256-dimensional feature extraction" → "225-dimensional feature extraction"

# EXCLUDE (keep as-is):
# - Model config files (d_model=256 is valid)
# - Non-feature uses (buffer sizes, layer dimensions)

Files to update (24 files, see Category 3 list above)

---

✅ Phase 3: Validation (2 hours)

Task 3.1: Run Updated Test Suite

# Test updated files
cargo test --test test_extract_256_dim_features -- --nocapture
cargo test --test dbn_256_feature_validation -- --nocapture
cargo test --test test_dbn_sequence_256_features -- --nocapture

# Expected output:
# ✅ All tests pass
# ✅ Feature vectors have 225 dimensions
# ✅ Real data validation succeeds

Task 3.2: Verify Feature Extraction with Real DBN Data

# Run full validation suite
cargo test --workspace --features "test-utils" | grep -i "feature"

# Expected:
# - 2,062/2,074 tests passing (99.4%)
# - No 256-dimension assertion failures
# - All feature extraction tests use 225 dimensions

Task 3.3: Update Documentation

# Update validation report
echo "Legacy 256-feature tests updated to 225 features" >> WAVE_D_VALIDATION_COMPLETE.md

# Update CLAUDE.md if needed
# Add note about legacy test cleanup completion

---

RISK ASSESSMENT

✅ Low Risk

• Feature extraction tests are isolated
• Wave D already validated with 225 features (99.4% test pass rate)
• Model config tests don't need changes (d_model=256 remains valid)
• No production code changes required

⚠️ Medium Risk

• Breaking feature extraction tests during update
• Mitigation:
  • Create branch: git checkout -b cleanup/legacy-256-tests
  • Update incrementally (one file at a time)
  • Run tests after each file change
  • Merge only after full validation

📊 Impact Analysis

Area	Impact	Mitigation
Production Deployment	None	Already validated with 225 features
ML Retraining	Blocked	Complete before Week 2-3 post-deployment
Test Suite	3 tests fail	Fix in 8-10 hours
Developer Confusion	Medium	Clear documentation in this file

---

TIMELINE

Phase 1: Critical Feature Tests (7 hours, sequential)
├─ Task 1.1: test_extract_256_dim_features.rs (2h)
├─ Task 1.2: dbn_256_feature_validation.rs (3h)
└─ Task 1.3: test_dbn_sequence_256_features.rs (2h)

Phase 2: Documentation Update (1 hour, parallel with Phase 1)
└─ Task 2.1: Global comment updates (1h)

Phase 3: Validation (2 hours, after Phase 1+2)
├─ Task 3.1: Run updated test suite (0.5h)
├─ Task 3.2: Verify with real DBN data (1h)
└─ Task 3.3: Update documentation (0.5h)

TOTAL: 8-10 hours (1-2 days, non-blocking)

---

PRIORITY JUSTIFICATION

MEDIUM Priority (Not Urgent, Important)

✅ Wave D production deployment NOT blocked:

• Current test suite: 99.4% pass rate (2,062/2,074)
• Production readiness: 92% (23/25 checkboxes)
• Wave D backtest: 7/7 tests passing (Sharpe 2.00, Win Rate 60%)
• 225-feature system already validated and operational

⚠️ Blocks future ML retraining:

• ML retraining roadmap: 4-6 weeks
• Requires 225-feature validation tests
• Legacy 256-feature tests will fail
• Creates confusion for new developers

📅 Recommended Timeline:

• Start: Week 2 after production deployment
• Complete: Week 3 after production deployment
• Before: ML model retraining (Week 4-10)

---

FILES REQUIRING UPDATES

🔴 Critical (MUST update before ML retraining)

1. /home/jgrusewski/Work/foxhunt/ml/tests/test_extract_256_dim_features.rs (206 lines)
2. /home/jgrusewski/Work/foxhunt/ml/tests/dbn_256_feature_validation.rs (616 lines)
3. /home/jgrusewski/Work/foxhunt/ml/tests/test_dbn_sequence_256_features.rs (387 lines)

🟡 Documentation (SHOULD update for clarity)

See Category 3 list (24 files with comment updates)

✅ No Changes Needed

• 18 model configuration test files (d_model parameter tests)
• Non-feature "256" uses (buffer sizes, layer dimensions)

---

VALIDATION CHECKLIST

After cleanup, verify:

• All 3 critical test files pass with 225 features
• Feature extraction produces 225-dimensional vectors
• DBN validation tests succeed for all symbols (ES.FUT, NQ.FUT, 6E.FUT, ZN.FUT)
• DbnSequenceLoader supports d_model=225 (default) and backward compatible with 256
• Test suite still at 99%+ pass rate
• No regression in model config tests (d_model=256 still works)
• Documentation updated (WAVE_D_VALIDATION_COMPLETE.md, CLAUDE.md)
• Git branch merged: cleanup/legacy-256-tests

---

APPENDIX: FULL FILE LIST (45 files)

Category 1: Critical Feature Tests (3 files)

1. ml/tests/test_extract_256_dim_features.rs
2. ml/tests/dbn_256_feature_validation.rs
3. ml/tests/test_dbn_sequence_256_features.rs

Category 2: Model Config Tests (18 files, NO CHANGES)

4. ml/tests/e2e_mamba2_training.rs
5. ml/tests/mamba_training_test.rs
6. ml/tests/mamba2_e2e_training.rs
7. ml/tests/mamba2_training_pipeline_test.rs
8. ml/tests/ensemble_4_model_trainable_integration.rs
9. ml/tests/tft_quantized_attention_unit_test.rs
10. ml/tests/streaming_pipeline_edge_cases.rs
11. ml/tests/ppo_e2e_training.rs
12. ml/tests/liquid_networks_test.rs
13. ml/tests/gpu_4_model_stress_test.rs
14. ml/tests/gpu_memory_budget_validation.rs
15. ml/tests/memory_optimization_tests.rs
16. ml/tests/test_dbn_parser_fix.rs
17. ml/tests/test_streaming_loader.rs
18. ml/tests/varmap_weight_extraction_test.rs
19. ml/tests/checkpoint_test.rs
20. ml/tests/model_registry_checkpoint_test.rs
21. ml/tests/tft_int8_calibration_dataset_test.rs

Category 3: Documentation (24 files, comment updates)

22. ml/tests/feature_cache_tests.rs
23. ml/tests/microstructure_tests.rs
24. ml/tests/alternative_bars_integration_test.rs
25. ml/tests/calibration_dataset_test.rs
26. ml/tests/dbn_feature_config_test.rs
27. ml/tests/e2e_ensemble_integration.rs
28. ml/tests/model_registry_tests.rs
29. ml/tests/model_validation_comprehensive.rs
30. ml/tests/ppo_gae_test.rs
31. ml/tests/ppo_training_pipeline_test.rs
32. ml/tests/test_feature_cache_service.rs
33. ml/tests/test_tft_cuda_layernorm.rs
34. ml/tests/tft_attention_gradient_flow.rs
35. ml/tests/tft_attention_int8_quantization_test.rs
36. ml/tests/tft_e2e_training.rs
37. ml/tests/tft_inference_latency_benchmark.rs
38. ml/tests/tft_int8_inference_integration_test.rs
39. ml/tests/tft_int8_memory_benchmark_test.rs
40. ml/tests/tft_varmap_checkpoint_test.rs
41. ml/tests/training_chaos_tests.rs
42. ml/tests/wave_d_normalization_integration_test.rs
43. ml/tests/mamba_test.rs
44. ml/tests/mamba2_hardware_aware_test.rs
45. ml/tests/ensemble_4_model_trainable_integration.rs (duplicate, in both categories)

Category 4: Non-Feature Uses (NO CHANGES)

• ml/tests/unsafe_validation_tests.rs
• Various files with non-feature "256" uses

---

NOTES

1. Backward Compatibility: Models MUST continue to support d_model=256 for flexibility and backward compatibility with existing checkpoints.

2. Test Strategy: Update feature extraction tests to use 225 as the default, but add tests for multiple d_model values (128, 225, 256, 512) to validate model flexibility.

3. No Production Impact: All production code already uses 225 features (Wave D implementation complete). This cleanup is purely for test suite consistency.

4. ML Retraining Dependency: Before retraining models with 225 features, these tests MUST pass to validate the feature extraction pipeline.

---

End of Report