foxhunt/DATABASE_INITIALIZATION_QUICK_REFERENCE.md

Database Initialization & Setup - Quick Reference

Last Updated: 2025-10-30 Purpose: Quick lookup for file locations and organizational structure

---

TL;DR: What Moves Where

KEEP IN ROOT (Do Not Move)

✅ init-db.sql                  # Database initialization (CRITICAL)
✅ init-db-dev.sql             # Dev database variant
✅ docker-compose.yml          # Service orchestration
✅ Dockerfile.foxhunt-build    # Production build
✅ tuning_config.yaml          # ML config (Docker mount)
✅ certs/, config/, models/    # Docker volume mounts

ARCHIVE TO artifacts/ (Analysis Files)

🗂️ artifacts/YYYY-MM-DD/
   ├── WAVE*.txt               # 200+ test result files
   ├── coverage_*.txt          # Test coverage reports
   ├── DB_LOAD_TEST_*.txt      # Database test results
   ├── *_SUMMARY.txt           # Agent work summaries
   ├── *_QUICKREF.txt          # Reference docs
   └── *.log files             # Build/test logs

MOVE TO docs/sql/diagnostics/ (SQL Test Files)

📋 docs/sql/diagnostics/
   ├── PAPER_TRADING_DIAGNOSTIC_QUERIES.sql
   ├── paper_trading_schema.sql
   ├── test_pg_performance.sql
   ├── test_stop_loss_debug.sql
   └── trading_workload.sql

MOVE TO docs/cargo-configs-archive/ (Build Config Variants)

⚙️ docs/cargo-configs-archive/
   ├── config.toml.coverage
   ├── config.toml.lld
   ├── config.toml.optimized
   ├── config.toml.original
   └── config.toml.runpod

👉 Keep: .cargo/config.toml (active config)

MOVE TO config/testing/ (Test Configurations)

🧪 config/testing/
   ├── pytest.ini
   ├── tarpaulin.toml
   └── mutants.toml

MOVE TO config/ml/tuning/archive/ (ML Config Backups)

🤖 config/ml/tuning/
   ├── archive/
   │   ├── tuning_config_ppo_comprehensive.yaml
   │   └── TFT_TUNING_CONFIG_RECOMMENDED.yaml

👉 Keep: tuning_config.yaml (active - Docker mount)

---

Docker Dependencies (CANNOT MOVE)

Path	Service	Docker Mount	Critical
certs/	All (TLS)	./certs:/tmp/foxhunt/certs:ro	🔒 YES
checkpoints/	ML Training	./checkpoints:/tmp/foxhunt/checkpoints	📦 YES
config/	Grafana, Prometheus	./config/grafana/*, ./config/prometheus/*	📊 YES
models/	ML Training	./models:/tmp/foxhunt/models	🤖 YES
optuna_studies/	ML Training	./optuna_studies:/app/optuna_studies	🔬 YES
test_data/	Backtesting	./test_data:/workspace/test_data:ro	📊 YES
tuning_config.yaml	ML Training	./tuning_config.yaml:/app/tuning_config.yaml:ro	⚙️ YES

If you move these, Docker WILL FAIL ❌

---

Current Root File Count

Category	Count	Status	Action
SQL Init Files	2	✅ Organized	Keep in root
Database Migrations	45	✅ Organized	Keep in migrations/
Config Files	10	⚠️ Scattered	Consolidate
Analysis/Test Results	400+	❌ Cluttered	Archive to artifacts/
Total Root Files	500+		Cleanup needed

---

File Organization Rules

Rule 1: Rust Standards (MUST BE IN ROOT)

✅ Cargo.toml               # Workspace manifest
✅ Cargo.lock               # Dependency lock
✅ .cargo/                  # Cargo config dir
✅ rustfmt.toml             # Code formatting
✅ clippy.toml              # Linting rules

These are Rust conventions. Do not move them.

Rule 2: Docker Dependencies (MUST BE IN ROOT)

✅ docker-compose.yml       # Service orchestration
✅ Dockerfile.*             # Container builds
✅ .dockerignore            # Docker exclusions
✅ init-db.sql              # Database setup
✅ certs/, config/, models/ # Volume mounts

Docker references these by relative path. Do not move them.

Rule 3: CI/CD Files (MUST BE IN ROOT)

✅ .gitlab-ci.yml           # GitLab pipeline
✅ .github/                 # GitHub workflows
✅ Makefile                 # Build targets

CI/CD systems look for these in root. Do not move them.

Rule 4: Everything Else CAN Move

⚠️ pytest.ini               # Can move to config/testing/
⚠️ tarpaulin.toml          # Can move to config/testing/
⚠️ *.txt analysis files    # Move to artifacts/
⚠️ sql/diagnostics         # Move to docs/sql/
⚠️ .cargo/config.toml.*    # Archive old variants

---

Migration Checklist

Pre-Migration

• Read DATABASE_INITIALIZATION_AND_SETUP_ANALYSIS.md (full report)
• Verify Docker is not running
• Backup docker-compose.yml before any changes
• Confirm no references to moved files in CI/CD

Migration Steps

Step 1: Archive Analysis Files (ZERO RISK)

mkdir -p artifacts/$(date +%Y-%m-%d)
find . -maxdepth 1 -name "*.txt" -type f \
  -exec mv {} artifacts/$(date +%Y-%m-%d)/ \;

Step 2: Move SQL Diagnostics (ZERO RISK)

mkdir -p docs/sql/diagnostics
mv sql/*.sql docs/sql/diagnostics/
rmdir sql

Step 3: Archive .cargo Variants (ZERO RISK)

mkdir -p docs/cargo-configs-archive
mv .cargo/config.toml.* docs/cargo-configs-archive/

Step 4: Optional - Move Testing Configs

mkdir -p config/testing
mv pytest.ini tarpaulin.toml mutants.toml config/testing/
# Update CI/CD references if using pytest/tarpaulin

Step 5: Optional - Move ML Tuning Backups

mkdir -p config/ml/tuning/archive
mv tuning_config_ppo_comprehensive.yaml config/ml/tuning/archive/
mv TFT_TUNING_CONFIG_RECOMMENDED.yaml config/ml/tuning/archive/
# Keep tuning_config.yaml in root

Post-Migration

• Run docker-compose config to verify syntax
• Test Docker startup: docker-compose up -d postgres && docker-compose logs -f
• Verify migrations apply: cargo sqlx migrate info
• Update documentation links
• Commit cleanup changes

---

Risk Assessment

Action	Risk	Notes
Archive .txt files	✅ ZERO	Just analysis, not code
Move SQL diagnostics	✅ ZERO	Dev-only files, not used by Docker
Archive .cargo variants	✅ ZERO	Keep main config, just archive old ones
Move test configs	⚠️ LOW	Update CI/CD references
Move Python requirements	⚠️ LOW-MED	Update pip install paths
Move tuning configs	⚠️ LOW	Keep main config in root (Docker mount)
Move Makefile targets	⚠️ MED	Update developer scripts

---

What NOT To Move

NEVER Move These (Breaks Docker)

❌ certs/                   # Docker mount dependency
❌ checkpoints/             # Docker mount dependency
❌ config/grafana/          # Docker mount dependency
❌ config/prometheus/       # Docker mount dependency
❌ models/                  # Docker mount dependency
❌ optuna_studies/          # Docker mount dependency
❌ test_data/               # Docker mount dependency
❌ tuning_config.yaml       # Docker mount dependency
❌ docker-compose.yml       # Docker uses this
❌ Dockerfile.*             # Docker uses this
❌ init-db.sql              # Database initialization

NEVER Move These (Breaks Rust/Cargo)

❌ Cargo.toml               # Workspace manifest
❌ Cargo.lock               # Dependency lock
❌ .cargo/                  # Cargo config dir
❌ rustfmt.toml             # Code formatting config
❌ clippy.toml              # Linting config

NEVER Move These (Breaks CI/CD)

❌ .gitlab-ci.yml           # GitLab pipeline
❌ .github/                 # GitHub workflows
❌ Makefile                 # Build targets
❌ .gitignore               # Git configuration

---

Current Structure Summary

Root Files by Category:
├── 🔒 Docker-Critical: 10 files (DO NOT MOVE)
├── ⚙️ Rust-Standard: 5 files (DO NOT MOVE)
├── 🔄 CI/CD: 5 files (DO NOT MOVE)
├── 📁 Source Directories: 10 dirs (ORGANIZED)
├── 🗂️ Volume Mounts: 8 dirs (DO NOT MOVE)
├── ⚠️ Can Consolidate: 10 files (MOVE SOON)
├── ❌ Clutter: 400+ .txt files (ARCHIVE NOW)
└── 📋 Legacy: 5+ SQL test files (MOVE NOW)

---

Key Dates/Versions

• init-db.sql: Last modified 2025-09-25 (5.2K)
• init-db-dev.sql: Last modified 2025-09-26 (659B)
• migrations/: 45 files, last: 046_batch_job_tracking.sql
• .cargo/config.toml: Last modified 2025-10-24 (active)
• docker-compose.yml: Current as of 2025-10-30
• Dockerfile.foxhunt-build: Last modified 2025-10-29

---

Commands Cheat Sheet

Verify Docker Setup

docker-compose config                   # Validate YAML
docker-compose ps                       # Running services
docker-compose logs -f postgres         # Check database

Database Setup

psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt
cargo sqlx migrate info                 # Show migration status
cargo sqlx migrate run                  # Apply pending migrations

Cleanup Commands

# Archive analysis files
mkdir -p artifacts/$(date +%Y-%m-%d)
mv *.txt artifacts/$(date +%Y-%m-%d)/

# Move SQL diagnostics
mkdir -p docs/sql/diagnostics
mv sql/*.sql docs/sql/diagnostics/

# Archive .cargo variants
mkdir -p docs/cargo-configs-archive
mv .cargo/config.toml.* docs/cargo-configs-archive/

---

Related Documentation

• Full Analysis: DATABASE_INITIALIZATION_AND_SETUP_ANALYSIS.md
• System Overview: CLAUDE.md
• Docker Guide: docker-compose.yml
• Migrations: migrations/ directory
• Scripts: scripts/README.md

---

FAQ

Q: Can I move init-db.sql? A: Not without checking how it's used. It's in root for a reason - likely referenced by Docker or setup scripts.

Q: Will moving files break Docker? A: YES - Only if you move files listed in the "NEVER MOVE" section. Check docker-compose.yml volume mounts first.

Q: Do I need to move everything? A: No. Priority 1 (archive .txt files) is the only critical cleanup. Rest is optional.

Q: How long does cleanup take? A: ~1-2 hours for full cleanup (Phase 1 + Phase 2). Priority 1 alone: 10-15 minutes.

Q: Will cleanup break the build? A: No - as long as you don't move Docker dependencies or Rust standard files.

---

Generated: 2025-10-30 Status: Ready to implement