foxhunt/LEGACY_256_TEST_CLEANUP.md

# LEGACY TEST CODE AUDIT: 256-Feature References

**Generated**: 2025-10-20
**Agent**: Legacy Test Code Audit
**Context**: Wave D complete with 225 features (201 Wave C + 24 Wave D). Legacy 256-feature test code documented for future cleanup.

---

## EXECUTIVE SUMMARY

```
Files found: 45
Total legacy test lines: 1,209 (core) + ~3,500 (documentation)
Total 256 references: 140 (feature/dimension context)
Cleanup effort: 8-12 hours
Priority: MEDIUM
```

**Status**: Wave D production deployment NOT blocked. Legacy tests need updating before ML retraining (4-6 week roadmap item).

---

## CATEGORY BREAKDOWN

### CATEGORY 1: CRITICAL - FEATURE EXTRACTION TESTS ⚠️
**3 files, 1,209 lines - MUST UPDATE**

These explicitly test 256-dimensional feature extraction and will FAIL with current 225-feature implementation:

#### 1. ml/tests/test_extract_256_dim_features.rs (206 lines)
```rust
// Current (BROKEN):
assert_eq!(feature_vec.len(), 256);

// Required:
assert_eq!(feature_vec.len(), 225);
```
- **Impact**: HIGH - Validates core feature extraction pipeline
- **Update effort**: 2 hours
- **Changes**: Assertions 256→225, test names, comments

#### 2. ml/tests/dbn_256_feature_validation.rs (616 lines)
```rust
// Current (BROKEN):
for feat_idx in 0..256 {
    assert_eq!(report.feature_stats.len(), 256);
}

// Required:
for feat_idx in 0..225 {
    assert_eq!(report.feature_stats.len(), 225);
}
```
- **Impact**: HIGH - Real data validation (ES.FUT, NQ.FUT, 6E.FUT, ZN.FUT)
- **Update effort**: 3 hours
- **Changes**: Loop bounds, assertions, statistical analysis

#### 3. ml/tests/test_dbn_sequence_256_features.rs (387 lines)
```rust
// Current (PARTIALLY BROKEN):
let mut loader = DbnSequenceLoader::with_limits(60, 256, Some(10), 10);

// Required (backward compatible):
let mut loader = DbnSequenceLoader::with_limits(60, 225, Some(10), 10);
// Also test: d_model in [128, 225, 256, 512]
```
- **Impact**: MEDIUM - Loader configuration test
- **Update effort**: 2 hours
- **Changes**: Default d_model=225, add multi-value tests

---

### CATEGORY 2: MODEL CONFIGURATION TESTS ✅
**18 files - NO CHANGES REQUIRED**

These test model architectures where `d_model=256` is a VALID configuration choice. Models support variable dimensions (128, 256, 512, etc.). DO NOT CHANGE.

**Files:**
1. ml/tests/e2e_mamba2_training.rs
2. ml/tests/mamba_training_test.rs
3. ml/tests/mamba2_e2e_training.rs
4. ml/tests/mamba2_training_pipeline_test.rs
5. ml/tests/ensemble_4_model_trainable_integration.rs
6. ml/tests/tft_quantized_attention_unit_test.rs
7. ml/tests/streaming_pipeline_edge_cases.rs
8. ml/tests/ppo_e2e_training.rs
9. ml/tests/liquid_networks_test.rs
10. ml/tests/gpu_4_model_stress_test.rs
11. ml/tests/gpu_memory_budget_validation.rs
12. ml/tests/memory_optimization_tests.rs
13. ml/tests/test_dbn_parser_fix.rs
14. ml/tests/test_streaming_loader.rs
15. ml/tests/varmap_weight_extraction_test.rs
16. ml/tests/checkpoint_test.rs
17. ml/tests/model_registry_checkpoint_test.rs
18. ml/tests/tft_int8_calibration_dataset_test.rs

**Rationale**: These tests validate model flexibility across different `d_model` values. The models MUST support 256 for backward compatibility and production flexibility.

---

### CATEGORY 3: DOCUMENTATION/COMMENTS 📝
**24 files, ~140 references - SHOULD UPDATE**

Files with "256" in comments/docs referring to old feature count. Low priority, easy fix.

**Update Strategy**: Global search-replace in comments
```bash
# Example:
# "256-dim features" → "225-dim features"
# "256 features per bar" → "225 features per bar"
# "256-feature pipeline" → "225-feature pipeline"
```

**Files:**
1. ml/tests/feature_cache_tests.rs (comments: "256-dim vectors")
2. ml/tests/microstructure_tests.rs (comment: "256-feature pipeline")
3. ml/tests/alternative_bars_integration_test.rs (comment: "256 features per bar")
4. ml/tests/calibration_dataset_test.rs (comment: "256 features if using full MAMBA-2")
5. ml/tests/dbn_feature_config_test.rs
6. ml/tests/e2e_ensemble_integration.rs
7. ml/tests/model_registry_tests.rs
8. ml/tests/model_validation_comprehensive.rs
9. ml/tests/ppo_gae_test.rs
10. ml/tests/ppo_training_pipeline_test.rs
11. ml/tests/test_feature_cache_service.rs
12. ml/tests/test_tft_cuda_layernorm.rs
13. ml/tests/tft_attention_gradient_flow.rs
14. ml/tests/tft_attention_int8_quantization_test.rs
15. ml/tests/tft_e2e_training.rs
16. ml/tests/tft_inference_latency_benchmark.rs
17. ml/tests/tft_int8_inference_integration_test.rs
18. ml/tests/tft_int8_memory_benchmark_test.rs
19. ml/tests/tft_varmap_checkpoint_test.rs
20. ml/tests/training_chaos_tests.rs
21. ml/tests/wave_d_normalization_integration_test.rs
22. ml/tests/mamba_test.rs
23. ml/tests/mamba2_hardware_aware_test.rs
24. ml/tests/ensemble_4_model_trainable_integration.rs

**Update effort**: 1 hour (automated search-replace)

---

### CATEGORY 4: NON-FEATURE USES ✅
**Files using "256" for non-feature purposes - NO CHANGES**

- ml/tests/unsafe_validation_tests.rs (`buffer.set_len(256)` - buffer capacity)
- ml/tests/liquid_networks_test.rs (`(128, 256)` - neural network layer sizes)
- ml/tests/mamba_training_test.rs (`Tensor::randn(&[1, 128, 256])` - shape tests)

**Rationale**: Not related to feature dimensions, DO NOT CHANGE.

---

## CLEANUP TASK LIST

### ✅ Phase 1: Critical Feature Tests (7 hours)

#### Task 1.1: Update test_extract_256_dim_features.rs (2h)
```bash
# File: ml/tests/test_extract_256_dim_features.rs
# Lines: 206
# Changes:
- [ ] Line 40-48: Change assertions 256 → 225
- [ ] Line 9: Rename test to test_extract_225_dim_features
- [ ] Line 1-4: Update file header documentation
- [ ] Verify with: cargo test --test test_extract_256_dim_features
```

**Key Changes:**
```rust
// Before:
assert_eq!(feature_vec.len(), 256, "Feature vector {} has wrong dimension", i);

// After:
assert_eq!(feature_vec.len(), 225, "Feature vector {} has wrong dimension", i);
```

#### Task 1.2: Update dbn_256_feature_validation.rs (3h)
```bash
# File: ml/tests/dbn_256_feature_validation.rs
# Lines: 616
# Changes:
- [ ] Line 1-21: Update file header (256 → 225)
- [ ] Line 172: Change loop bounds (0..256 → 0..225)
- [ ] Line 230, 242, 268, 294: Update assertions (256 → 225)
- [ ] Line 313, 331, 359: Update feature count assertions
- [ ] Line 411: Update performance targets
- [ ] Verify with: cargo test --test dbn_256_feature_validation
```

**Key Changes:**
```rust
// Before:
let mut feature_stats = Vec::with_capacity(256);
for feat_idx in 0..256 { /* ... */ }
assert_eq!(report.feature_stats.len(), 256);

// After:
let mut feature_stats = Vec::with_capacity(225);
for feat_idx in 0..225 { /* ... */ }
assert_eq!(report.feature_stats.len(), 225);
```

#### Task 1.3: Update test_dbn_sequence_256_features.rs (2h)
```bash
# File: ml/tests/test_dbn_sequence_256_features.rs
# Lines: 387
# Changes:
- [ ] Line 1-4: Update file header (256 → 225)
- [ ] Line 38: Change default d_model (256 → 225)
- [ ] Line 69, 98: Update shape documentation
- [ ] Line 237-259: Add multi-d_model test (128, 225, 256, 512)
- [ ] Verify backward compatibility
- [ ] Verify with: cargo test --test test_dbn_sequence_256_features
```

**Key Changes:**
```rust
// Before:
let mut loader = DbnSequenceLoader::with_limits(60, 256, Some(10), 10);
println!("✅ Created DbnSequenceLoader (seq_len=60, d_model=256, max=10, stride=10)\n");

// After (with backward compatibility test):
let mut loader = DbnSequenceLoader::with_limits(60, 225, Some(10), 10);
println!("✅ Created DbnSequenceLoader (seq_len=60, d_model=225, max=10, stride=10)\n");

// Add new test:
#[tokio::test]
async fn test_multiple_d_model_values() -> Result<()> {
    for d_model in [128, 225, 256, 512] {
        let mut loader = DbnSequenceLoader::with_limits(60, d_model, Some(10), 10).await?;
        // ... validate each d_model ...
    }
}
```

---

### ✅ Phase 2: Documentation Update (1 hour)

#### Task 2.1: Global Comment Update
```bash
# Search patterns:
grep -r "256.*feature" ml/tests/ --include="*.rs"
grep -r "256.*dim" ml/tests/ --include="*.rs"
grep -r "256-dimensional" ml/tests/ --include="*.rs"

# Replace (manual review required):
# "256-dim features" → "225-dim features"
# "256 features per bar" → "225 features per bar"
# "256-feature pipeline" → "225-feature pipeline"
# "256-dimensional feature extraction" → "225-dimensional feature extraction"

# EXCLUDE (keep as-is):
# - Model config files (d_model=256 is valid)
# - Non-feature uses (buffer sizes, layer dimensions)
```

**Files to update** (24 files, see Category 3 list above)

---

### ✅ Phase 3: Validation (2 hours)

#### Task 3.1: Run Updated Test Suite
```bash
# Test updated files
cargo test --test test_extract_256_dim_features -- --nocapture
cargo test --test dbn_256_feature_validation -- --nocapture
cargo test --test test_dbn_sequence_256_features -- --nocapture

# Expected output:
# ✅ All tests pass
# ✅ Feature vectors have 225 dimensions
# ✅ Real data validation succeeds
```

#### Task 3.2: Verify Feature Extraction with Real DBN Data
```bash
# Run full validation suite
cargo test --workspace --features "test-utils" | grep -i "feature"

# Expected:
# - 2,062/2,074 tests passing (99.4%)
# - No 256-dimension assertion failures
# - All feature extraction tests use 225 dimensions
```

#### Task 3.3: Update Documentation
```bash
# Update validation report
echo "Legacy 256-feature tests updated to 225 features" >> WAVE_D_VALIDATION_COMPLETE.md

# Update CLAUDE.md if needed
# Add note about legacy test cleanup completion
```

---

## RISK ASSESSMENT

### ✅ Low Risk
- Feature extraction tests are isolated
- Wave D already validated with 225 features (99.4% test pass rate)
- Model config tests don't need changes (d_model=256 remains valid)
- No production code changes required

### ⚠️ Medium Risk
- Breaking feature extraction tests during update
- **Mitigation**:
  - Create branch: `git checkout -b cleanup/legacy-256-tests`
  - Update incrementally (one file at a time)
  - Run tests after each file change
  - Merge only after full validation

### 📊 Impact Analysis
| Area | Impact | Mitigation |
|------|--------|------------|
| Production Deployment | None | Already validated with 225 features |
| ML Retraining | Blocked | Complete before Week 2-3 post-deployment |
| Test Suite | 3 tests fail | Fix in 8-10 hours |
| Developer Confusion | Medium | Clear documentation in this file |

---

## TIMELINE

```
Phase 1: Critical Feature Tests (7 hours, sequential)
├─ Task 1.1: test_extract_256_dim_features.rs (2h)
├─ Task 1.2: dbn_256_feature_validation.rs (3h)
└─ Task 1.3: test_dbn_sequence_256_features.rs (2h)

Phase 2: Documentation Update (1 hour, parallel with Phase 1)
└─ Task 2.1: Global comment updates (1h)

Phase 3: Validation (2 hours, after Phase 1+2)
├─ Task 3.1: Run updated test suite (0.5h)
├─ Task 3.2: Verify with real DBN data (1h)
└─ Task 3.3: Update documentation (0.5h)

TOTAL: 8-10 hours (1-2 days, non-blocking)
```

---

## PRIORITY JUSTIFICATION

### MEDIUM Priority (Not Urgent, Important)

**✅ Wave D production deployment NOT blocked:**
- Current test suite: 99.4% pass rate (2,062/2,074)
- Production readiness: 92% (23/25 checkboxes)
- Wave D backtest: 7/7 tests passing (Sharpe 2.00, Win Rate 60%)
- 225-feature system already validated and operational

**⚠️ Blocks future ML retraining:**
- ML retraining roadmap: 4-6 weeks
- Requires 225-feature validation tests
- Legacy 256-feature tests will fail
- Creates confusion for new developers

**📅 Recommended Timeline:**
- **Start**: Week 2 after production deployment
- **Complete**: Week 3 after production deployment
- **Before**: ML model retraining (Week 4-10)

---

## FILES REQUIRING UPDATES

### 🔴 Critical (MUST update before ML retraining)
1. `/home/jgrusewski/Work/foxhunt/ml/tests/test_extract_256_dim_features.rs` (206 lines)
2. `/home/jgrusewski/Work/foxhunt/ml/tests/dbn_256_feature_validation.rs` (616 lines)
3. `/home/jgrusewski/Work/foxhunt/ml/tests/test_dbn_sequence_256_features.rs` (387 lines)

### 🟡 Documentation (SHOULD update for clarity)
See Category 3 list (24 files with comment updates)

### ✅ No Changes Needed
- 18 model configuration test files (d_model parameter tests)
- Non-feature "256" uses (buffer sizes, layer dimensions)

---

## VALIDATION CHECKLIST

After cleanup, verify:

- [ ] All 3 critical test files pass with 225 features
- [ ] Feature extraction produces 225-dimensional vectors
- [ ] DBN validation tests succeed for all symbols (ES.FUT, NQ.FUT, 6E.FUT, ZN.FUT)
- [ ] DbnSequenceLoader supports d_model=225 (default) and backward compatible with 256
- [ ] Test suite still at 99%+ pass rate
- [ ] No regression in model config tests (d_model=256 still works)
- [ ] Documentation updated (WAVE_D_VALIDATION_COMPLETE.md, CLAUDE.md)
- [ ] Git branch merged: `cleanup/legacy-256-tests`

---

## APPENDIX: FULL FILE LIST (45 files)

### Category 1: Critical Feature Tests (3 files)
1. ml/tests/test_extract_256_dim_features.rs
2. ml/tests/dbn_256_feature_validation.rs
3. ml/tests/test_dbn_sequence_256_features.rs

### Category 2: Model Config Tests (18 files, NO CHANGES)
4. ml/tests/e2e_mamba2_training.rs
5. ml/tests/mamba_training_test.rs
6. ml/tests/mamba2_e2e_training.rs
7. ml/tests/mamba2_training_pipeline_test.rs
8. ml/tests/ensemble_4_model_trainable_integration.rs
9. ml/tests/tft_quantized_attention_unit_test.rs
10. ml/tests/streaming_pipeline_edge_cases.rs
11. ml/tests/ppo_e2e_training.rs
12. ml/tests/liquid_networks_test.rs
13. ml/tests/gpu_4_model_stress_test.rs
14. ml/tests/gpu_memory_budget_validation.rs
15. ml/tests/memory_optimization_tests.rs
16. ml/tests/test_dbn_parser_fix.rs
17. ml/tests/test_streaming_loader.rs
18. ml/tests/varmap_weight_extraction_test.rs
19. ml/tests/checkpoint_test.rs
20. ml/tests/model_registry_checkpoint_test.rs
21. ml/tests/tft_int8_calibration_dataset_test.rs

### Category 3: Documentation (24 files, comment updates)
22. ml/tests/feature_cache_tests.rs
23. ml/tests/microstructure_tests.rs
24. ml/tests/alternative_bars_integration_test.rs
25. ml/tests/calibration_dataset_test.rs
26. ml/tests/dbn_feature_config_test.rs
27. ml/tests/e2e_ensemble_integration.rs
28. ml/tests/model_registry_tests.rs
29. ml/tests/model_validation_comprehensive.rs
30. ml/tests/ppo_gae_test.rs
31. ml/tests/ppo_training_pipeline_test.rs
32. ml/tests/test_feature_cache_service.rs
33. ml/tests/test_tft_cuda_layernorm.rs
34. ml/tests/tft_attention_gradient_flow.rs
35. ml/tests/tft_attention_int8_quantization_test.rs
36. ml/tests/tft_e2e_training.rs
37. ml/tests/tft_inference_latency_benchmark.rs
38. ml/tests/tft_int8_inference_integration_test.rs
39. ml/tests/tft_int8_memory_benchmark_test.rs
40. ml/tests/tft_varmap_checkpoint_test.rs
41. ml/tests/training_chaos_tests.rs
42. ml/tests/wave_d_normalization_integration_test.rs
43. ml/tests/mamba_test.rs
44. ml/tests/mamba2_hardware_aware_test.rs
45. ml/tests/ensemble_4_model_trainable_integration.rs (duplicate, in both categories)

### Category 4: Non-Feature Uses (NO CHANGES)
- ml/tests/unsafe_validation_tests.rs
- Various files with non-feature "256" uses

---

## NOTES

1. **Backward Compatibility**: Models MUST continue to support d_model=256 for flexibility and backward compatibility with existing checkpoints.

2. **Test Strategy**: Update feature extraction tests to use 225 as the default, but add tests for multiple d_model values (128, 225, 256, 512) to validate model flexibility.

3. **No Production Impact**: All production code already uses 225 features (Wave D implementation complete). This cleanup is purely for test suite consistency.

4. **ML Retraining Dependency**: Before retraining models with 225 features, these tests MUST pass to validate the feature extraction pipeline.

---

**End of Report**