docs: add stub detection & prevention design
Approved design for full-codebase audit using 6 parallel agents scanning 14 stub patterns, plus lint prevention in pre-commit hook. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
108
docs/plans/2026-02-24-stub-detection-design.md
Normal file
108
docs/plans/2026-02-24-stub-detection-design.md
Normal file
@@ -0,0 +1,108 @@
|
||||
# Stub Detection & Prevention Design
|
||||
|
||||
**Date**: 2026-02-24
|
||||
**Status**: Approved
|
||||
**Goal**: Find and fix all fake/stub code hiding behind passing tests, then prevent new stubs from entering the codebase.
|
||||
|
||||
## Problem
|
||||
|
||||
The foxhunt codebase (37+ crates, ~200K lines of production Rust) contains stub code that compiles and passes tests but doesn't implement real logic. Examples already found and fixed:
|
||||
|
||||
- Kelly `get_historical_returns()` returned `sin(x)` instead of real market data
|
||||
- `get_current_market_price()` hardcoded `100.0`
|
||||
- PPO hyperopt used random actions with fake log_probs
|
||||
- DQN/PPO validation computed market volatility instead of model performance
|
||||
- ML orders always returned BUY regardless of prediction
|
||||
- Risk metrics returned hardcoded `0.0` values
|
||||
|
||||
These stubs are dangerous because they **compile, pass tests, and look real at a glance**.
|
||||
|
||||
## Approach
|
||||
|
||||
Two-phase strategy:
|
||||
1. **Agent swarm audit** — one-time deep scan across all crates, fix everything possible
|
||||
2. **Lint prevention** — add pattern-based checks to pre-commit hook and CI
|
||||
|
||||
## Phase 1: Stub Pattern Taxonomy (14 patterns)
|
||||
|
||||
### Grep-Detectable
|
||||
|
||||
| # | Pattern | Detection |
|
||||
|---|---------|-----------|
|
||||
| 1 | Hardcoded returns | `return 100.0`, `return vec![]`, `return 0.0` in non-test code |
|
||||
| 2 | Placeholder strings | `"TODO"`, `"placeholder"`, `"fixme"`, `"stub"` in string literals |
|
||||
| 3 | Random in production | `rand::` usage in `src/` (not tests/benchmarks/examples) |
|
||||
| 4 | Dead integration points | `unimplemented`, `Status::unimplemented`, `StatusCode::NOT_IMPLEMENTED` |
|
||||
| 5 | Discarded real values | `_` prefixed variables from important return tuples |
|
||||
| 6 | Config value shadowing | Function params that are never read (shadowed by hardcoded values) |
|
||||
| 7 | Feature-gated stubs | `#[cfg(feature = "...")]` where default branch is a stub |
|
||||
| 8 | Always-same match arms | All match arms returning the same variant |
|
||||
|
||||
### Semantic (Agent-Analyzed)
|
||||
|
||||
| # | Pattern | Detection |
|
||||
|---|---------|-----------|
|
||||
| 9 | Fake data generators | Math functions (`sin`, `cos`) in data retrieval methods |
|
||||
| 10 | Wrong metric | Function name vs. what it actually computes (e.g., "validation" computing volatility) |
|
||||
| 11 | Zeroed structs | `..Default::default()` where financial fields should be computed |
|
||||
| 12 | Passthrough no-ops | Functions that return input unchanged in pipeline stages |
|
||||
| 13 | Test-shaped production | Production code that only works for test scenarios |
|
||||
| 14 | Stub with matching test | Test assertions that match function's hardcoded returns |
|
||||
|
||||
## Phase 1: Agent Swarm Architecture
|
||||
|
||||
6 parallel agents, each responsible for a crate group:
|
||||
|
||||
| Agent | Crates | Focus |
|
||||
|-------|--------|-------|
|
||||
| ML Agent | `ml/`, `ml-data/` | Model training, inference, hyperopt |
|
||||
| Risk Agent | `risk/`, `risk-data/` | Position sizing, VaR, kill switches |
|
||||
| Trading Agent | `trading_engine/`, `adaptive-strategy/` | Execution, ensemble, microstructure |
|
||||
| Services Agent | `services/` (all 8) | gRPC endpoints, integration points |
|
||||
| Infra Agent | `common/`, `config/`, `data/`, `database/`, `storage/` | Shared utilities, data loading |
|
||||
| Frontend Agent | `web-gateway/`, `tli/`, `backtesting/` | API routes, CLI, backtesting |
|
||||
|
||||
### Agent Protocol
|
||||
|
||||
Each agent:
|
||||
1. **Grep scan** — run all 8 grep-detectable patterns
|
||||
2. **Semantic scan** — read flagged functions, assess if real stubs
|
||||
3. **Classify** — severity (Critical/Important/Minor) + confidence (High/Medium/Low)
|
||||
4. **Fix** — auto-fix where real implementation is clear
|
||||
5. **Report** — structured findings list with file, line, severity, fix status
|
||||
|
||||
### Severity Classification
|
||||
|
||||
- **Critical**: Affects trading decisions, risk calculations, or model training (must fix)
|
||||
- **Important**: Affects data quality, metrics accuracy, or integration correctness (should fix)
|
||||
- **Minor**: Cosmetic stubs, unused code paths, or deprecated features (can defer)
|
||||
|
||||
## Phase 2: Lint Prevention Layer
|
||||
|
||||
Add to `scripts/pre-commit-check.sh`:
|
||||
|
||||
### New Checks
|
||||
|
||||
1. **Hardcoded returns** in non-test `.rs` files: `return 0\.0`, `return 100\.0`, `return vec![]`
|
||||
2. **Stub markers**: `todo!`, `unimplemented!`, `placeholder`, `hardcoded`, `FIXME`
|
||||
3. **Production randomness**: `rand::` in `src/` (not tests/benches/examples)
|
||||
4. **Clippy**: `#[deny(clippy::unused_self)]` for passthrough no-ops
|
||||
|
||||
### Behavior
|
||||
|
||||
- **Pre-commit**: Warn on pattern matches (non-blocking)
|
||||
- **CI**: Block on Critical patterns in `ml/`, `risk/`, `trading_engine/`, `services/`
|
||||
|
||||
## Follow-up: TLI Rename
|
||||
|
||||
Separate commit to rename `tli/` → `cli/`:
|
||||
- Rename crate directory
|
||||
- Update `Cargo.toml` workspace members
|
||||
- Update `[[bin]]` target name
|
||||
- Update imports/references in services
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- Zero Critical stubs remaining in production code paths
|
||||
- Pre-commit hook catches new stub patterns
|
||||
- Each finding has a classification (severity + confidence) and either a fix or documented justification
|
||||
Reference in New Issue
Block a user