foxhunt/docs/archive/historical/DOCUMENTATION_RESTRUCTURE.md

# Documentation Restructure - 2025-10-06

## Changes Made

### 1. CLAUDE.md - Rewritten for Architecture & Fundamentals ✅

**OLD Focus**: Progress tracking, wave history, status updates
**NEW Focus**: Architecture fundamentals, infrastructure, credentials, how to use existing components

**Key Sections Added**:
- 🏗️ **Service Topology Diagram**: Visual architecture map
- 🔑 **Infrastructure & Credentials**: Database connection strings from docker-compose.yml
  - PostgreSQL: `postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt`
  - Redis: `redis://localhost:6379`
  - InfluxDB: `foxhunt:foxhunt_dev_password`
  - Vault: `foxhunt-dev-root`
  - Grafana: `admin:foxhunt123`
  - Prometheus: `localhost:9090`
- 📁 **Codebase Structure**: Clear directory layout with purposes
- 🛠️ **Development Workflow**: Initial setup, common commands, running services
- 📞 **Quick Reference**: Docker, database ops, health checks, coverage

**Emphasis on REUSE**:
- ✅ Parquet Market Data Replay (existing)
- ✅ Backtesting Service gRPC API (existing)
- ✅ Feature Engineering Pipeline (existing)
- ✅ Docker infrastructure (docker-compose.yml)
- ❌ DO NOT rebuild components

**Core Principle Highlighted**:
> **REUSE existing infrastructure. DO NOT rebuild components.**

---

### 2. TESTING_PLAN.md - New Comprehensive ML Testing Strategy ✅

**Created**: Standalone testing plan for ML/AI validation with realistic crypto data

**Key Sections**:

1. **Existing Infrastructure (REUSE)**: 90% already implemented
   - ParquetMarketDataWriter (production-ready)
   - BacktestingService with gRPC
   - Feature engineering pipeline
   - ParquetMarketDataReader (INCOMPLETE - needs implementation)

2. **Required Additions**: Only 3 components needed
   - Complete ParquetMarketDataReader (2-4 hours)
   - Binance WebSocket client (4-6 hours)
   - Test datasets generation (2-3 hours)

3. **4-Tier Testing Strategy**:
   - **Tier 1**: Unit tests with mocks (30 min runtime)
   - **Tier 2**: Integration tests with 1-hour Parquet replay
   - **Tier 3**: Multi-regime backtesting with 1-week dataset
   - **Tier 4**: Live simulation (future work)

4. **Implementation Timeline**:
   - **Week 1**: Complete ParquetReader + datasets
   - **Week 2**: Binance client + multi-regime data
   - **Week 3**: ML validation tests (DQN, MAMBA-2, TFT, Liquid)

5. **Infrastructure Setup**: Detailed database credentials and setup commands

**Expected Impact**: +15-20% coverage (51% → 70%)

---

### 3. WAVE Files Cleanup ✅

**BEFORE**: 219 WAVE report files
**AFTER**: 3 essential summary files

**Kept**:
- `WAVE112_FINAL_STATUS.md` - Systematic compilation fix
- `WAVE113_FINAL_SUMMARY.md` - Coverage unblocking & security
- `WAVE114_FINAL_REPORT.md` - Service compilation fixes

**Deleted**: 216 files
- All individual agent reports (WAVE*_AGENT*.md)
- Historical waves 30-111
- Redundant planning/certification documents
- Duplicate summaries

**Rationale**: Keep only the final, comprehensive reports for the 3 most recent waves.

---

## Documentation Structure (Now)

```
foxhunt/
├── CLAUDE.md                      # Architecture fundamentals & infrastructure
├── TESTING_PLAN.md                # ML testing strategy (NEW)
├── README.md                      # Project overview
├── .env.example                   # Environment template
├── docker-compose.yml             # Infrastructure (credentials source)
├── WAVE112_FINAL_STATUS.md        # Wave 112 summary
├── WAVE113_FINAL_SUMMARY.md       # Wave 113 summary
├── WAVE114_FINAL_REPORT.md        # Wave 114 summary
├── DOCUMENTATION_RESTRUCTURE.md   # This file
└── migrations/README.md           # Database schema docs
```

---

## Key Improvements

### 1. Credentials are Now Accessible ✅

**Before**: No clear documentation of database credentials
**After**: All credentials documented in CLAUDE.md from docker-compose.yml

```bash
# PostgreSQL
postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt

# Redis
redis://localhost:6379

# Vault
http://localhost:8200 (token: foxhunt-dev-root)

# Grafana
http://localhost:3000 (admin:foxhunt123)
```

### 2. Infrastructure Reuse is Emphasized ✅

**CLAUDE.md** includes "REUSE" sections:
- 🧪 Testing Infrastructure (REUSE)
- 📞 Quick Reference for existing services
- 🚫 Anti-Workaround Protocol with "REUSE" examples

**TESTING_PLAN.md** is built entirely around existing infrastructure:
- Existing Components: 90% complete
- Required Additions: 3 small components
- Anti-Patterns section: "DO NOT rebuild"

### 3. Architecture is Front and Center ✅

**Service Topology Diagram**:
```
TLI → API Gateway → (Trading, Backtesting, ML Training) → (PostgreSQL, Redis)
```

**Component Responsibilities**: Clear ownership and boundaries
**Codebase Structure**: Directory layout with purposes
**Service Ports**: External vs internal port mapping

### 4. Reduced Clutter ✅

**WAVE Files**: 219 → 3 (98.6% reduction)
- Easier to navigate
- Focus on essential summaries
- Historical context preserved in kept files

---

## Migration Guide for Claude Sessions

### For New Sessions

1. **Start with CLAUDE.md**: Architecture, credentials, infrastructure
2. **Reference TESTING_PLAN.md**: For ML/AI testing strategy
3. **Check Recent Waves**: WAVE114_FINAL_REPORT.md for latest status

### For Ongoing Work

- Use `docker-compose up -d` to start infrastructure
- Database URL: `postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt`
- Always REUSE existing components (see TESTING_PLAN.md)

### For Testing

- TESTING_PLAN.md has complete strategy
- Database credentials in CLAUDE.md
- Existing infrastructure sections in both files

---

## Summary

**CLAUDE.md**: Progress tracker → Architecture & fundamentals guide
**TESTING_PLAN.md**: Created with crypto integration strategy
**WAVE files**: 219 → 3 essential summaries
**Focus**: Emphasize REUSE of existing infrastructure

**Next Steps**:
1. Use TESTING_PLAN.md to implement ML testing (Wave 115)
2. Reference CLAUDE.md for infrastructure and credentials
3. Historical context available in 3 WAVE summaries

---

**Created**: 2025-10-06
**Files Modified**: 2 (CLAUDE.md rewritten, TESTING_PLAN.md created)
**Files Deleted**: 216 obsolete WAVE reports
**Files Kept**: 3 essential WAVE summaries