foxhunt/ROOT_CONFIG_FILES_ANALYSIS_REPORT.md

Root Configuration Files Analysis - Foxhunt HFT System

Analysis Date: 2025-10-30 Repository: /home/jgrusewski/Work/foxhunt Total Root Config Files: 20+ files

---

INVENTORY: Configuration Files in Root

A. ENVIRONMENT FILES (.env*)

File	Size	Status	Type	Purpose
.env	1.6KB	Git-ignored	ACTIVE	Local dev credentials (JWT, DB, Redis, Databento)
.env.example	6.6KB	Tracked	Template	Comprehensive template with AWS S3, Vault, DataBento
.env.runpod	887B	Git-ignored	ACTIVE	RunPod API, S3, GPU config (Oct 24)
.env.runpod.template	3.4KB	Tracked	Template	RunPod template with instructions

Key Differences:

• .env (actual): 40 lines, minimal config (JWT, DB, Redis only)
• .env.example (template): 172 lines, comprehensive (S3, Vault, retention, lifecycle)
• .env.runpod (actual): 25 lines, production RunPod credentials
• .env.runpod.template (template): 86 lines, detailed with documentation

---

B. DOCKER COMPOSE FILES

File	Size	Git Status	Type	Purpose
docker-compose.yml	17KB	Tracked ✅	Production	4 services: API Gateway, Trading, Backtesting, ML Training
docker-compose.override.yml	627B	Tracked ✅	Development	Local overrides (env vars, volumes)

Status: Both essential, well-integrated

---

C. BUILD & TOOLING CONFIGS

File	Size	Git Status	Type	Purpose
Cargo.toml	19KB	Tracked ✅	Root Workspace	Workspace config, shared dependencies
clippy.toml	582B	Tracked ✅	Rust Linting	Clippy configuration
rustfmt.toml	7.3KB	Tracked ✅	Rust Formatting	Code style rules
tarpaulin.toml	1.4KB	Tracked ✅	Coverage	Code coverage tool config
mutants.toml	2.5KB	Tracked ✅	Mutation Testing	Mutation test settings
pytest.ini	-	Git-ignored ⚠️	Python Testing	pytest configuration (NEW)

Issues:

• ✅ All properly tracked except pytest.ini (should document)

---

D. MACHINE LEARNING TUNING CONFIGS

File	Size	Lines	Git Status	Usage
tuning_config.yaml	430B	26	Tracked ✅	Minimal test config for TLI tune command
tuning_config_ppo_comprehensive.yaml	5.7KB	184	Tracked ✅	PPO hyperparameter tuning (Epoch 380 baseline)
TFT_TUNING_CONFIG_RECOMMENDED.yaml	10KB	258	Tracked ✅	TFT-FP32 recommended tuning (60% cache speedup)

Analysis:

• All three are referenced in code (tli/src/commands/tune.rs default value: tuning_config.yaml)
• Code references: ml/examples/tune_hyperparameters.rs, services/ml_training_service/tests/
• Should STAY in root - used as default configs for ML training examples
• Alternative location: config/ml/tuning/*.yaml (but breaks default path assumptions)

In config/ml/ already:

• ✅ config/ml/ subdirectory exists with ML-specific configs
• ⚠️ Tuning configs in root are easier to access for examples/docs

---

E. DOCUMENTATION FILES (ROOT .md/.txt)

Extremely Large Root Documentation (30+ markdown files):

Category	Count	Sample Size	Status
CLAUDE.md	1	12KB	Essential ✅
CI/CD Guides	4+	8-47KB	Docker, GitLab, Local CI
RunPod Guides	4+	5-21KB	Deployment, workflow, quick refs
Checklists	6+	4-35KB	Deployment, security, training
Legacy Reports	100+	1-20KB	Agent summaries, test results

Major Findings:

Root documentation volume:
- .md files: 30+ files
- .txt files: 100+ temporary/legacy files (>1GB total)
- Legacy files: agent summaries, test logs, coverage reports

---

F. LEGACY/REPORT FILES (Should Archive)

Pattern	Count	Total Size	Last Modified	Status
WAVE_*.txt	20+	~500MB	Oct 2025	Legacy reports
AGENT*.txt	30+	~200MB	Oct 2025	Agent deliverables
*_results.txt	15+	~1.5GB	Oct 2025	Test/build logs
*_SUMMARY.txt	10+	~50MB	Oct 2025	Executive summaries
.coverage, coverage.xml	2	~5MB	Oct 2025	Coverage artifacts

Examples:

• clippy_results.txt: 1MB
• final_test_results.txt: 377KB
• WAVE_141_FULL_TEST_RESULTS.txt: 279KB
• coverage_output.txt: 79KB

---

ANALYSIS: What Should Stay/Move/Delete

STAY IN ROOT (Essential Infrastructure)

Environment Files (4 files - 12KB total):

• ✅ .env - Dev credentials (git-ignored)
• ✅ .env.example - Template (tracked)
• ✅ .env.runpod - RunPod credentials (git-ignored)
• ✅ .env.runpod.template - RunPod template (tracked)
• Reason: Standard practice for any deployment system; needed on startup

Docker Compose (2 files):

• ✅ docker-compose.yml - Production services
• ✅ docker-compose.override.yml - Local dev overrides
• Reason: Docker standard location; referenced by build/deploy scripts

Build Configuration (6 files):

• ✅ Cargo.toml - Rust workspace
• ✅ clippy.toml - Linting
• ✅ rustfmt.toml - Code format
• ✅ tarpaulin.toml - Coverage
• ✅ mutants.toml - Mutation testing
• ✅ pytest.ini - Python tests (new)
• Reason: Build tool standard locations

Master Documentation:

• ✅ CLAUDE.md - System architecture & status (12KB)
• ✅ README.md - Project overview (20KB)
• Reason: Essential entry point for developers

---

RELOCATE (Can Move to config/ or docs/)

ML Tuning Configs (3 files - 16KB total):

RECOMMENDATION: Keep in root, BUT:
- Create symlinks from config/ml/tuning/ → root
- Document in README that defaults live in root
- Alternative: Move to config/ml/tuning/ and update code default path

Current state (BETTER):
  tuning_config.yaml → tli tune --config tuning_config.yaml

If moved (BREAKS DEFAULTS):
  config/ml/tuning/tuning_config.yaml → tli tune --config config/ml/tuning/tuning_config.yaml

Decision: KEEP IN ROOT - used as defaults in examples

Quick Reference Guides (Move to docs/):

• DOCKER_BUILD_GUIDE.md (7.4KB)
• DOCKER_BUILD_QUICK_REF.md (13KB)
• GITLAB_CI_QUICK_REF.md (8.4KB)
• LOCAL_CI_PIPELINE_GUIDE.md (9.8KB)
• MONITOR_LOGS_QUICK_REF.md (8.9KB)
• RUNPOD_DEPLOY_QUICK_REF.md (4.5KB)
• RUNPOD_WORKFLOW_GUIDE.md (21KB)
• HYPEROPT_DEPLOYMENT_GUIDE.md (21KB)
• OOM_RECOVERY_GUIDE.md (25KB)
• etc.

Relocate Strategy:

Root (CURRENT - 15+ guides):
  DOCKER_BUILD_GUIDE.md
  GITLAB_CI_QUICK_REF.md
  RUNPOD_DEPLOYMENT_READINESS_CHECKLIST.md
  etc.

Better location:
  docs/deployment/DOCKER_BUILD_GUIDE.md
  docs/deployment/GITLAB_CI_QUICK_REF.md
  docs/runpod/DEPLOYMENT_READINESS_CHECKLIST.md
  docs/quickref/MONITOR_LOGS.md

---

DELETE (Cleanup - Legacy/Generated)

Temporary Test/Build Artifacts (~2.5GB total):

DELETE IMMEDIATELY:
  .coverage (binary coverage file)
  coverage.xml (coverage report)
  clippy_results.txt (1MB)
  final_clippy_results.txt (1MB)
  final_test_results.txt (377KB)
  ml_final_test.txt (122KB)
  full_test_results.txt (377KB)
  coverage_output.txt (79KB)
  build_log.txt, build_results.txt

DELETE - ARCHIVE FIRST:
  WAVE_*.txt files (20+ files, 500MB) - Old agent reports
  AGENT*.txt files (30+ files, 200MB) - Old deliverables
  *_SUMMARY.txt files - Legacy summaries
  *_results.txt files (except coverage.xml)
  final_clippy_*.txt files
  tft_*.txt files
  ppo_*.txt files
  ml_*.txt files
  trading_engine_test_output.txt
  hyperparameter_tuning_*.txt

Orphaned Config Files:

• Delete? Need to check references:
  • pytest.ini - NEW (Oct 30), should KEEP
  • .gitignore.python - Check if used
  • All others have clear purposes

---

RECOMMENDATIONS

1. IMMEDIATE CLEANUP (30 minutes)

# Archive legacy files to backup
tar czf foxhunt-legacy-reports-$(date +%Y%m%d).tar.gz \
  WAVE_*.txt AGENT*.txt *_SUMMARY.txt

# Delete temporary artifacts
rm -f .coverage coverage.xml
rm -f clippy_results.txt final_clippy_results.txt
rm -f final_test_results.txt ml_final_test.txt
rm -f *_results.txt (except coverage.xml)
rm -f build_log.txt build_results.txt build_*.txt
rm -f tft_*.txt ppo_*.txt ml_*.txt
rm -f *_output.txt (except production essentials)

Result: ~2GB freed, root cleaner

2. DOCUMENTATION REORGANIZATION (1-2 hours)

Current Structure:
  Root:
    CLAUDE.md (12KB) ✅
    README.md (20KB) ✅
    DOCKER_BUILD_GUIDE.md (7.4KB) ⬇️
    GITLAB_CI_QUICK_REF.md (8.4KB) ⬇️
    RUNPOD_WORKFLOW_GUIDE.md (21KB) ⬇️
    HYPEROPT_DEPLOYMENT_GUIDE.md (21KB) ⬇️
    + 20+ more guides...

Proposed Structure:
  docs/
    README.md → 'Documentation Index'
    architecture/
      CLAUDE.md (symlink to root)
    deployment/
      DOCKER_BUILD_GUIDE.md
      DOCKER_MULTISTAGE_PRODUCTION_GUIDE.md
      GITLAB_CI_DOCKER_SETUP_GUIDE.md
    quickref/
      DOCKER_BUILD_QUICK_REF.md
      GITLAB_CI_QUICK_REF.md
      MONITOR_LOGS_QUICK_REF.md
      BINARY_UPLOAD_QUICK_REF.md
    runpod/
      RUNPOD_WORKFLOW_GUIDE.md
      RUNPOD_DEPLOY_QUICK_REF.md
      RUNPOD_PYTHON_QUICK_REF.md
      RUNPOD_DEPLOYMENT_READINESS_CHECKLIST.md
    ml/
      HYPEROPT_DEPLOYMENT_GUIDE.md
      OOM_RECOVERY_GUIDE.md
      DQN_TRAINING_PATHS_QUICK_REF.md
    security/
      SECURITY_PRODUCTION_DEPLOYMENT_CHECKLIST.md
      SECURITY_HARDENING_CHECKLIST.md
    checklists/
      PRODUCTION_DEPLOYMENT_CHECKLIST.md
      PRE_FLIGHT_CHECKLIST.md
      PRE_DEPLOYMENT_CHECKLIST.md

3. ENVIRONMENT FILE ORGANIZATION

Current (Good):

.env                    ← Actual dev config (git-ignored)
.env.example            ← Template reference (tracked)
.env.runpod             ← Actual RunPod config (git-ignored)
.env.runpod.template    ← Template reference (tracked)

Keep as-is - This follows best practices.

Optional Enhancement:

Create docs/ENV_SETUP.md with:
  - Which .env files to use when
  - How to generate JWT_SECRET
  - RunPod credentials location guide
  - Quick comparison table

4. ML TUNING CONFIG STRATEGY

DECISION: Keep in root (reasons below)

Current (GOOD):
  tuning_config.yaml                    ← Used by: tli tune
  tuning_config_ppo_comprehensive.yaml  ← PPO training reference
  TFT_TUNING_CONFIG_RECOMMENDED.yaml    ← TFT training reference

Code dependencies:
  tli/src/commands/tune.rs:
    default_value = "tuning_config.yaml"  ← expects root location

  ml/examples/tune_hyperparameters.rs:
    // Define search ranges based on tuning_config.yaml (in comments)

  services/ml_training_service/tests/:
    create_real_tuning_config(&config_path)  ← dynamic path

Why keep in root:

1. Default path in tli tune command
2. Documentation examples reference root
3. Easy access for quick testing
4. Standard practice for config templates

If space is critical: Symlink from config/ml/tuning/

ln -s ../tuning_config.yaml config/ml/tuning/
ln -s ../tuning_config_ppo_comprehensive.yaml config/ml/tuning/
ln -s ../TFT_TUNING_CONFIG_RECOMMENDED.yaml config/ml/tuning/

---

FINAL SUMMARY TABLE

Category	Action	Files	Size	Impact
Keep in Root	✅ KEEP	.env*, docker-compose*, Cargo.toml, clippy.toml, etc.	75KB	Immediate startup
Keep (ML Tuning)	✅ KEEP	tuning_config*.yaml, TFT_TUNING_CONFIG*.yaml	16KB	Training defaults
Keep (Master Docs)	✅ KEEP	CLAUDE.md, README.md	32KB	Entry points
Move to docs/	➡️ MOVE	15+ guides (DOCKER_, GITLAB_, RUNPOD_*, etc.)	~300KB	Organization
Delete (Artifacts)	❌ DELETE	.coverage, coverage.xml, results.txt, build.txt	~2.5GB	Cleanup
Archive (Legacy)	<EFBFBD>️ ARCHIVE	WAVE_.txt, AGENT.txt, *_SUMMARY.txt	~700MB	Historical

---

GITIGNORE VERIFICATION

Current .gitignore (Mostly Correct):

.env          ✅ Ignored (credentials)
.env.*        ✅ Ignored (all .env variants except explicit exceptions)
!.env.example ✅ Tracked (template)
!.env.runpod  ⚠️ This should probably be !.env.runpod.template
!.env.runpod.template ✅ Tracked (template)

Issue Found: .gitignore has !.env.runpod but that's the ACTUAL credentials file (would be git-ignored by parent rule). The intent appears to be tracking the template only.

Recommended Fix:

.env.*                    # Ignore all .env.* files
!.env.example             # Except .env.example (template)
!.env.runpod.template     # Except .env.runpod.template (template)

# Remove this if present:
# !.env.runpod              ← This allows credentials to be tracked (incorrect)

---

FILES SAFE TO DELETE NOW

1. Coverage artifacts: .coverage, coverage.xml (regenerated by tests)
2. Build logs: build_log.txt, build_results.txt, clippy_*.txt
3. Test output: final_test_results.txt, ml_final_test.txt, *_output.txt
4. Legacy reports: All WAVE_*.txt and AGENT*.txt (archive first)
5. Temporary outputs: *_results.txt, *_output.txt (except documented exceptions)

Do NOT delete:

• Any .env* files (credentials)
• docker-compose*.yml files
• Cargo.toml and build configs
• Any .md files (keep CLAUDE.md, README.md, and guides for now)

---

IMPLEMENTATION PHASES

Phase 1: Quick Wins (15 minutes)

1. Delete .coverage and coverage.xml
2. Delete build_log.txt, build_results.txt
3. Delete clippy_results.txt, final_clippy_results.txt

Phase 2: Archive Legacy (20 minutes)

1. Create archives/ directory
2. Archive WAVE_*.txt, AGENT*.txt, *_SUMMARY.txt
3. Verify archive integrity
4. Delete originals

Phase 3: Organize Documentation (1 hour)

1. Create docs/ directory structure
2. Move guides to appropriate subdirectories
3. Update links in README/CLAUDE.md
4. Create docs/README.md index

Phase 4: Verify (10 minutes)

1. Run tests to ensure no broken paths
2. Verify CI/CD still works
3. Check that examples still run

---

REFERENCES & CONTEXT

Related Files:

• /home/jgrusewski/Work/foxhunt/.gitignore - Git ignore rules
• /home/jgrusewski/Work/foxhunt/.env - Active configuration
• /home/jgrusewski/Work/foxhunt/CLAUDE.md - System status
• /home/jgrusewski/Work/foxhunt/README.md - Project overview

External Standards:

• Docker: docker-compose.yml in root (standard)
• Rust: Cargo.toml in root (standard)
• Python: .env in root (standard)
• Environment variables: Template pattern is industry best practice