foxhunt/docs/WAVE102_AGENT9_COVERAGE_FIX.md

WAVE 102 AGENT 9: Filesystem Corruption Fix & Coverage Tool Recovery

Mission: Resolve filesystem issues preventing coverage measurement Date: 2025-10-04 Status: ✅ SOLUTION FOUND - Root cause identified and fixed

---

🔍 Root Cause Analysis

Previous Diagnosis: "Filesystem corruption blocking coverage tools" Actual Root Cause: Incompatible Rust compiler flags in .cargo/config.toml

The Issue

Wave 81 reported that both cargo-tarpaulin and cargo-llvm-cov were failing with:

• tarpaulin: error: unknown codegen option: stack-protector
• llvm-cov: "Filesystem corruption in target/ directory"

The real issue was NOT filesystem corruption - it was an incompatible compiler flag.

Investigation Results

1. ZFS Filesystem Health: ✅ PERFECT

   • Pool status: ONLINE
   • Scrub status: 0 errors
   • Disk space: 517GB free (81% available)
   • No corruption detected

2. Build Artifacts: ✅ INTACT

   • 13,585 .rmeta/.rlib files in target/
   • Build cache functioning normally
   • No filesystem-level issues

3. Compiler Flags: ❌ ROOT CAUSE FOUND

   • .cargo/config.toml line 12: -C stack-protector=strong
   • This flag is not supported by Rust 1.89.0 stable
   • Coverage tools add additional flags that conflict with this

---

🛠️ Solution

Step 1: Remove Incompatible Flag

File: .cargo/config.toml

Original Configuration (Line 12):

[build]
rustflags = [
    "-D", "unsafe_op_in_unsafe_fn",
    "-D", "clippy::undocumented_unsafe_blocks",
    "-W", "rust_2024_idioms",
    "-C", "force-frame-pointers=yes",
    "-C", "stack-protector=strong",  # <-- INCOMPATIBLE
    "-C", "relocation-model=pic",
]

Fixed Configuration:

[build]
rustflags = [
    "-D", "unsafe_op_in_unsafe_fn",
    "-D", "clippy::undocumented_unsafe_blocks",
    "-W", "rust_2024_idioms",
    "-C", "force-frame-pointers=yes",
    # REMOVED: "-C", "stack-protector=strong",  # Not compatible with coverage tools
    "-C", "relocation-model=pic",
]

Step 2: Clean and Rebuild

# Backup original config
cp .cargo/config.toml .cargo/config.toml.original

# Apply coverage-compatible config
cp .cargo/config.toml.coverage .cargo/config.toml

# Clean build artifacts to avoid flag conflicts
cargo clean

Step 3: Run Coverage Tools

cargo-llvm-cov (RECOMMENDED):

# Single crate
cargo llvm-cov --package common --html --output-dir target/coverage --ignore-run-fail

# JSON output for parsing
cargo llvm-cov --package common --json --output-path target/coverage/common.json --ignore-run-fail

# Workspace (may have compilation errors in some tests)
cargo llvm-cov --workspace --html --output-dir target/coverage --ignore-run-fail

cargo-tarpaulin (ALTERNATIVE):

cargo tarpaulin --workspace --timeout 300 --skip-clean --out Html --output-dir target/coverage

---

✅ Verification Results

Coverage Tool Status: ✅ OPERATIONAL

cargo-llvm-cov: ✅ WORKING

• Successfully generates HTML reports
• Successfully generates JSON reports
• --ignore-run-fail flag allows coverage despite test failures

cargo-tarpaulin: ✅ WORKING (after flag fix)

• No longer fails with "unknown codegen option"
• Generates coverage reports

Sample Coverage Data (common crate)

{
  "lines": 24.709854399662376,
  "functions": 33.43701399688958,
  "regions": 27.914642609299097
}

Coverage Metrics:

• Line coverage: 24.7%
• Function coverage: 33.4%
• Region coverage: 27.9%

---

📊 Impact Assessment

What Was NOT Wrong

❌ Filesystem corruption - ZFS pool is healthy with 0 errors ❌ Disk space exhaustion - 517GB free (81% available) ❌ File handle exhaustion - Well below system limits ❌ Parallel build race conditions - Not the root cause

What WAS Wrong

✅ Incompatible compiler flag - -C stack-protector=strong not supported ✅ Coverage tool conflicts - Tools add their own flags that clash ✅ Build configuration issue - Not a filesystem or infrastructure problem

---

🎯 Recommendations

Immediate Actions

1. Keep Coverage-Compatible Config for Testing

   • Use .cargo/config.toml.coverage when running coverage tools
   • Restore .cargo/config.toml.original for production builds

2. Document Configuration Switching

   # Switch to coverage config
   cp .cargo/config.toml.coverage .cargo/config.toml
   
   # Run coverage
   cargo llvm-cov --workspace --html --output-dir target/coverage --ignore-run-fail
   
   # Restore original config
   cp .cargo/config.toml.original .cargo/config.toml
   

3. Create Coverage Script

   • Automate config switching
   • Run coverage on all compilable crates
   • Generate comprehensive reports

Long-Term Solutions

1. Upgrade Rust Version

   • Once stack-protector flag is stable, can use in coverage
   • Track Rust 1.90+ for flag stabilization

2. Separate Build Profiles

   • Create [profile.coverage] in Cargo.toml
   • Apply different flags per profile
   • Avoid global rustflags conflicts

3. CI/CD Integration

   • Use coverage-compatible config in CI
   • Automate coverage reporting
   • Track coverage trends over time

---

📋 Files Modified

1. Created: .cargo/config.toml.coverage (coverage-compatible config)
2. Backed up: .cargo/config.toml.original (original config)
3. Modified: .cargo/config.toml (currently using coverage config)

---

🎯 Success Criteria: ✅ ALL MET

• [✅] Root cause identified: Incompatible -C stack-protector=strong flag
• [✅] Solution implemented: Coverage-compatible config created
• [✅] cargo-llvm-cov working: Generates HTML and JSON reports
• [✅] cargo-tarpaulin working: No longer fails with codegen error
• [✅] Coverage measurable: Successfully extracted metrics from common crate
• [✅] Documentation created: Comprehensive troubleshooting guide

---

📊 Coverage Measurement Status

Before Wave 102: ❌ BLOCKED - "Filesystem corruption" After Wave 102: ✅ OPERATIONAL - Coverage tools working

Method: cargo-llvm-cov with --ignore-run-fail flag Output Formats: HTML reports, JSON data Blockers Remaining: Test compilation errors (not coverage tool issues)

---

🚀 Next Steps

Wave 102 Remaining Work

1. Fix Test Compilation Errors

   • api_gateway: 4 errors in MFA tests
   • data: Type mismatches in provider tests
   • trading_service: 3 errors in execution tests
   • adaptive-strategy: Compilation errors in backtesting tests

2. Measure Workspace Coverage

   • Once tests compile, run: cargo llvm-cov --workspace --ignore-run-fail
   • Target: Validate 75-85% coverage estimate from Wave 81

3. Generate Comprehensive Reports

   • HTML reports for visual inspection
   • JSON data for automated analysis
   • Coverage trends tracking

---

📝 Conclusion

Problem: "Filesystem corruption blocking coverage tools" Reality: Incompatible compiler flag causing tool failures Solution: Remove -C stack-protector=strong for coverage runs Status: ✅ Coverage tools now operational

The "filesystem corruption" diagnosis was a red herring. The actual issue was a simple configuration conflict that prevented coverage tools from running. With the incompatible flag removed, both cargo-llvm-cov and cargo-tarpaulin work correctly.

---

Documentation created: 2025-10-04 Coverage Tools Status: ✅ OPERATIONAL Root Cause: Compiler flag incompatibility (NOT filesystem corruption)