foxhunt/DATABASE_INITIALIZATION_QUICK_REFERENCE.md

# Database Initialization & Setup - Quick Reference

**Last Updated**: 2025-10-30
**Purpose**: Quick lookup for file locations and organizational structure

---

## TL;DR: What Moves Where

### KEEP IN ROOT (Do Not Move)

```
✅ init-db.sql                  # Database initialization (CRITICAL)
✅ init-db-dev.sql             # Dev database variant
✅ docker-compose.yml          # Service orchestration
✅ Dockerfile.foxhunt-build    # Production build
✅ tuning_config.yaml          # ML config (Docker mount)
✅ certs/, config/, models/    # Docker volume mounts
```

### ARCHIVE TO artifacts/ (Analysis Files)

```
🗂️ artifacts/YYYY-MM-DD/
   ├── WAVE*.txt               # 200+ test result files
   ├── coverage_*.txt          # Test coverage reports
   ├── DB_LOAD_TEST_*.txt      # Database test results
   ├── *_SUMMARY.txt           # Agent work summaries
   ├── *_QUICKREF.txt          # Reference docs
   └── *.log files             # Build/test logs
```

### MOVE TO docs/sql/diagnostics/ (SQL Test Files)

```
📋 docs/sql/diagnostics/
   ├── PAPER_TRADING_DIAGNOSTIC_QUERIES.sql
   ├── paper_trading_schema.sql
   ├── test_pg_performance.sql
   ├── test_stop_loss_debug.sql
   └── trading_workload.sql
```

### MOVE TO docs/cargo-configs-archive/ (Build Config Variants)

```
⚙️ docs/cargo-configs-archive/
   ├── config.toml.coverage
   ├── config.toml.lld
   ├── config.toml.optimized
   ├── config.toml.original
   └── config.toml.runpod

👉 Keep: .cargo/config.toml (active config)
```

### MOVE TO config/testing/ (Test Configurations)

```
🧪 config/testing/
   ├── pytest.ini
   ├── tarpaulin.toml
   └── mutants.toml
```

### MOVE TO config/ml/tuning/archive/ (ML Config Backups)

```
🤖 config/ml/tuning/
   ├── archive/
   │   ├── tuning_config_ppo_comprehensive.yaml
   │   └── TFT_TUNING_CONFIG_RECOMMENDED.yaml

👉 Keep: tuning_config.yaml (active - Docker mount)
```

---

## Docker Dependencies (CANNOT MOVE)

| Path | Service | Docker Mount | Critical |
|------|---------|--------------|----------|
| `certs/` | All (TLS) | `./certs:/tmp/foxhunt/certs:ro` | 🔒 YES |
| `checkpoints/` | ML Training | `./checkpoints:/tmp/foxhunt/checkpoints` | 📦 YES |
| `config/` | Grafana, Prometheus | `./config/grafana/*`, `./config/prometheus/*` | 📊 YES |
| `models/` | ML Training | `./models:/tmp/foxhunt/models` | 🤖 YES |
| `optuna_studies/` | ML Training | `./optuna_studies:/app/optuna_studies` | 🔬 YES |
| `test_data/` | Backtesting | `./test_data:/workspace/test_data:ro` | 📊 YES |
| `tuning_config.yaml` | ML Training | `./tuning_config.yaml:/app/tuning_config.yaml:ro` | ⚙️ YES |

**If you move these, Docker WILL FAIL** ❌

---

## Current Root File Count

| Category | Count | Status | Action |
|----------|-------|--------|--------|
| SQL Init Files | 2 | ✅ Organized | Keep in root |
| Database Migrations | 45 | ✅ Organized | Keep in migrations/ |
| Config Files | 10 | ⚠️ Scattered | Consolidate |
| Analysis/Test Results | 400+ | ❌ Cluttered | Archive to artifacts/ |
| **Total Root Files** | **500+** | | **Cleanup needed** |

---

## File Organization Rules

### Rule 1: Rust Standards (MUST BE IN ROOT)

```
✅ Cargo.toml               # Workspace manifest
✅ Cargo.lock               # Dependency lock
✅ .cargo/                  # Cargo config dir
✅ rustfmt.toml             # Code formatting
✅ clippy.toml              # Linting rules
```

These are Rust conventions. Do not move them.

### Rule 2: Docker Dependencies (MUST BE IN ROOT)

```
✅ docker-compose.yml       # Service orchestration
✅ Dockerfile.*             # Container builds
✅ .dockerignore            # Docker exclusions
✅ init-db.sql              # Database setup
✅ certs/, config/, models/ # Volume mounts
```

Docker references these by relative path. Do not move them.

### Rule 3: CI/CD Files (MUST BE IN ROOT)

```
✅ .gitlab-ci.yml           # GitLab pipeline
✅ .github/                 # GitHub workflows
✅ Makefile                 # Build targets
```

CI/CD systems look for these in root. Do not move them.

### Rule 4: Everything Else CAN Move

```
⚠️ pytest.ini               # Can move to config/testing/
⚠️ tarpaulin.toml          # Can move to config/testing/
⚠️ *.txt analysis files    # Move to artifacts/
⚠️ sql/diagnostics         # Move to docs/sql/
⚠️ .cargo/config.toml.*    # Archive old variants
```

---

## Migration Checklist

### Pre-Migration

- [ ] Read `DATABASE_INITIALIZATION_AND_SETUP_ANALYSIS.md` (full report)
- [ ] Verify Docker is not running
- [ ] Backup `docker-compose.yml` before any changes
- [ ] Confirm no references to moved files in CI/CD

### Migration Steps

#### Step 1: Archive Analysis Files (ZERO RISK)
```bash
mkdir -p artifacts/$(date +%Y-%m-%d)
find . -maxdepth 1 -name "*.txt" -type f \
  -exec mv {} artifacts/$(date +%Y-%m-%d)/ \;
```

#### Step 2: Move SQL Diagnostics (ZERO RISK)
```bash
mkdir -p docs/sql/diagnostics
mv sql/*.sql docs/sql/diagnostics/
rmdir sql
```

#### Step 3: Archive .cargo Variants (ZERO RISK)
```bash
mkdir -p docs/cargo-configs-archive
mv .cargo/config.toml.* docs/cargo-configs-archive/
```

#### Step 4: Optional - Move Testing Configs
```bash
mkdir -p config/testing
mv pytest.ini tarpaulin.toml mutants.toml config/testing/
# Update CI/CD references if using pytest/tarpaulin
```

#### Step 5: Optional - Move ML Tuning Backups
```bash
mkdir -p config/ml/tuning/archive
mv tuning_config_ppo_comprehensive.yaml config/ml/tuning/archive/
mv TFT_TUNING_CONFIG_RECOMMENDED.yaml config/ml/tuning/archive/
# Keep tuning_config.yaml in root
```

### Post-Migration

- [ ] Run `docker-compose config` to verify syntax
- [ ] Test Docker startup: `docker-compose up -d postgres && docker-compose logs -f`
- [ ] Verify migrations apply: `cargo sqlx migrate info`
- [ ] Update documentation links
- [ ] Commit cleanup changes

---

## Risk Assessment

| Action | Risk | Notes |
|--------|------|-------|
| Archive `.txt` files | ✅ ZERO | Just analysis, not code |
| Move SQL diagnostics | ✅ ZERO | Dev-only files, not used by Docker |
| Archive `.cargo` variants | ✅ ZERO | Keep main config, just archive old ones |
| Move test configs | ⚠️ LOW | Update CI/CD references |
| Move Python requirements | ⚠️ LOW-MED | Update pip install paths |
| Move tuning configs | ⚠️ LOW | Keep main config in root (Docker mount) |
| Move Makefile targets | ⚠️ MED | Update developer scripts |

---

## What NOT To Move

### NEVER Move These (Breaks Docker)

```
❌ certs/                   # Docker mount dependency
❌ checkpoints/             # Docker mount dependency
❌ config/grafana/          # Docker mount dependency
❌ config/prometheus/       # Docker mount dependency
❌ models/                  # Docker mount dependency
❌ optuna_studies/          # Docker mount dependency
❌ test_data/               # Docker mount dependency
❌ tuning_config.yaml       # Docker mount dependency
❌ docker-compose.yml       # Docker uses this
❌ Dockerfile.*             # Docker uses this
❌ init-db.sql              # Database initialization
```

### NEVER Move These (Breaks Rust/Cargo)

```
❌ Cargo.toml               # Workspace manifest
❌ Cargo.lock               # Dependency lock
❌ .cargo/                  # Cargo config dir
❌ rustfmt.toml             # Code formatting config
❌ clippy.toml              # Linting config
```

### NEVER Move These (Breaks CI/CD)

```
❌ .gitlab-ci.yml           # GitLab pipeline
❌ .github/                 # GitHub workflows
❌ Makefile                 # Build targets
❌ .gitignore               # Git configuration
```

---

## Current Structure Summary

```
Root Files by Category:
├── 🔒 Docker-Critical: 10 files (DO NOT MOVE)
├── ⚙️ Rust-Standard: 5 files (DO NOT MOVE)
├── 🔄 CI/CD: 5 files (DO NOT MOVE)
├── 📁 Source Directories: 10 dirs (ORGANIZED)
├── 🗂️ Volume Mounts: 8 dirs (DO NOT MOVE)
├── ⚠️ Can Consolidate: 10 files (MOVE SOON)
├── ❌ Clutter: 400+ .txt files (ARCHIVE NOW)
└── 📋 Legacy: 5+ SQL test files (MOVE NOW)
```

---

## Key Dates/Versions

- **init-db.sql**: Last modified 2025-09-25 (5.2K)
- **init-db-dev.sql**: Last modified 2025-09-26 (659B)
- **migrations/**: 45 files, last: 046_batch_job_tracking.sql
- **.cargo/config.toml**: Last modified 2025-10-24 (active)
- **docker-compose.yml**: Current as of 2025-10-30
- **Dockerfile.foxhunt-build**: Last modified 2025-10-29

---

## Commands Cheat Sheet

### Verify Docker Setup
```bash
docker-compose config                   # Validate YAML
docker-compose ps                       # Running services
docker-compose logs -f postgres         # Check database
```

### Database Setup
```bash
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt
cargo sqlx migrate info                 # Show migration status
cargo sqlx migrate run                  # Apply pending migrations
```

### Cleanup Commands
```bash
# Archive analysis files
mkdir -p artifacts/$(date +%Y-%m-%d)
mv *.txt artifacts/$(date +%Y-%m-%d)/

# Move SQL diagnostics
mkdir -p docs/sql/diagnostics
mv sql/*.sql docs/sql/diagnostics/

# Archive .cargo variants
mkdir -p docs/cargo-configs-archive
mv .cargo/config.toml.* docs/cargo-configs-archive/
```

---

## Related Documentation

- **Full Analysis**: `DATABASE_INITIALIZATION_AND_SETUP_ANALYSIS.md`
- **System Overview**: `CLAUDE.md`
- **Docker Guide**: `docker-compose.yml`
- **Migrations**: `migrations/` directory
- **Scripts**: `scripts/README.md`

---

## FAQ

**Q: Can I move init-db.sql?**
A: Not without checking how it's used. It's in root for a reason - likely referenced by Docker or setup scripts.

**Q: Will moving files break Docker?**
A: YES - Only if you move files listed in the "NEVER MOVE" section. Check `docker-compose.yml` volume mounts first.

**Q: Do I need to move everything?**
A: No. Priority 1 (archive `.txt` files) is the only critical cleanup. Rest is optional.

**Q: How long does cleanup take?**
A: ~1-2 hours for full cleanup (Phase 1 + Phase 2). Priority 1 alone: 10-15 minutes.

**Q: Will cleanup break the build?**
A: No - as long as you don't move Docker dependencies or Rust standard files.

---

**Generated**: 2025-10-30
**Status**: Ready to implement